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SUMMARY 


This  report  describes  the  procedures  employed  in  the  application  of 
instructional  Systems  Development  (ISD)  in  the  E-2  aircrew  training  community. 
Phase  I,  which  involved  the  analysis  and  design  study,  is  sumnarized.  Phase  II, 
which  involved  development,  implementation  and  evaluation,  is  described  in 
detail  along  with  an  evaluation  of  the  program's  effectiveness.  Additionally, 
a  Follow-on  Revision  effort  is  documented . 

The  model  which  provided  the  direction  for  this  effort  was  developed  by 
the  contractor  in  consonance  with  the  Systems  Approach  to  Training  (SAT) . 

The  Statement -of -Work  provided  by  the  Naval  Training  Equipment  Center  followed 
the  traditional  approach  to  ISD.  The  Phase  II  and  Follow-on  Revision  efforts 
were  conducted  jointly  by  the  Navy’s  E-2  training  squadrons  (RVAW-120  and 
RVAW-110)  and  the  contractor. 

Outputs  of  the  ISD  effort  to  date  include  fully  implemented  and  revised 
E-2C  syllabi  for  pilot  and  NFO/FT  training  and  E-2B  pilo.  training.  Additionally, 
mediated  lessons  covering  ISD  procedures  were  developed  by  the  contractor  for 
generic  as  well  as  E-2  utility. 

Results  of  the  evaluation  indicate  that  ISD  has  produced  an  E-2  aircrew 
training  program  which  is  both  valid  and  cost-effective.  Substantial  reductions 
in  course  length  have  been  achieved,  while  training  effectiveness  appears  to 
have  been  significantly  improved. 
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PREFACE 


The  work  discussed  in  this  report  represents  the  culmination  of  the 
formal  development,  implementation,  evaluation  and  revision  of  the  E-2 
Aircrew  Training  Program.  This  effort  was  one  of  four  such  projects 
(A-6E,  EA-6B,  E-2B/E-2C  and  SH-2F  Weapons  Systems)  begun  in  early  1975 
by  the  Naval  Training  Equipment  Center  for  the  Naval  Air  Systems  Command, 
to  develop  aircrew  training  systems  and  to  establish  the  requirements  for 
implementing  the  systems  approach  to  training  within  the  Naval  aviation 
community.  This  program  was  the  second  of  the  four  to  go  on-line  and  fleet 
feedback  indicates  a  marked  improvement  in  the  overall  quality  of  the  program 
graduates. 

Appreciation  is  expressed  to  those  personnel  of  RVAW-120,  RVAW-110, 
COMNAVAIRLANT,  COMNAVAIRPAC ,  and  COMCAEWWING  TWELVE  who  made  significant 
contributions  to  this  project. 


ifcU  J£), 

ROBERT  G.  BIRD 


Acquisition  Directors 
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SECTION  I 


INTRODUCTION 


PROGRAM  GOALS  AND  CONSTRAINTS 

The  Naval  Training  Equipment  Center  (NTEC)  is  currently  involved  in  a 
comprehensive  program  of  Instructional  Systems  Development  (ISD)  within  the 
aviation  training  community.  The  overall  program  began  in  March  of  1974 
when  NTEC  awarded  a  contract  for  the  design  and  development  of  an  aircrew 
training  program  for  the  SH-2F  LAMPS  aircraft.  This  was  followed  in 
April  with  similar  contracts  for  the  A-6E  TRAM,  EA-6B  ICAP  and  E-2C 
weapon  systems.  The  contracts  called  for  a  two-phased  effort.  Phase  I 
involved  the  development  of  Task  Analyses,  Behavioral  Objectives,  Train,  .g 
Support  Requirements  (media),  and  Lesson  Specifications.  Phase  II  involved 
Instructional  Materials  Development,  Instructor  Training,  Implementation, 
Evaluation,  and  Revision.  The  E-2  ISD  program  is  one  of  two  such  programs 
which  have  completed  the  Phase  II  effort  to  date.  While  both  programs  have 
been  developed  within  the  guidelines  of  the  ISD  framework,  each  has  used 
divergent  methodologies  to  achieve  the  end-products  and  each  has  been 
adapted  to  the  unique  requirements  of  differing  aircrew  training  systems, 
missions,  and  squadrons.  The  purpose  of  this  Training  Program  Work  Report 
is  to  document  the  procedures  employed  in  Phase  II  of  the  E-2  ISD  effort, 
to  examine  their  effectiveness,  and  to  evaluate  the  outcome  of  the  overall 
training  program. 

NTEC's  project  outline  (N215-263)  for  the  Phase  II  effort  specified 
the  following  objectives: 

a.  To  produce  an  initial  version  of  aircrew  training  materials 
for  E-2C  aircrews; 

b.  To  supplement  the  instructional  materials  with  appropriate 
media ; 

c.  To  implement  the  complete  program  at  RVAW-I20,  NAS  Norfolk; 

d.  To  evaluate  the  effectiveness  of  the  program  and  generate 
specifications  for  needed  revisions  ; 

e.  To  revise  the  programs  according  to  the  specifications; 

f.  To  develop  an  in-depth  instructor  training  program; 

g.  To  tabulate  manpower  and  cost  data  for  future  planning 
purposes  ;  and 

h.  To  document  the  Phase  II  effort. 
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The  outline  further  delineated  that  the  project  would  include  the  following 
tasks : 

(1)  Instructional  Materials  Development; 

(2)  Evaluation  Planning; 

(3)  Implementation  Planning; 

(4)  Instructor  Training  Course  Development; 

(5)  Implementation  and  Evaluation; 

(6)  Revision;  and 

(7)  Resource  Utilization  Data  • 

Prior  to  award  of  the  contract,  the  scope  of  the  program  was  expanded  to 
include  the  adaptation  of  the  E-2C  effort  to  E-2B  pilot/copilot  training  as  well 
Additionally,  the  task  of  incorporating  all  E-2C  ISD  documentation  into  a 
computer  data  base  for  the  Aviation  Training  Support  System  (ATSS)  was  specified 

Preparation  for  Phase  II  of  the  E-2  ISD  effort  began  in  April  of  1976 
with  the  project  specification  (N215-263)  which  was  followed  by  delivery 
of  the  contractor's  proposal  in  June.  Proposal  Review  Meetings  were  then 
held  at  NAVAIR  on  21  July  and  at  RVAW-120  on  28  July.  One  out  ome  of  these 
meetings  was  the  Navy's  approval  of  the  contractor's  Task  Analyses,  Behavioral 
Objectives  and  Lesson  Specifications  from  the  Phase  I  effort.  A  second  out¬ 
come  was  a  decision  to  heavily  integrate  Naval  personnel  into  the  Phase  II 
development  process,  using  RVAW-120  E-2C  Subject  Matter  Experts  (SME's)  as 
authors  of  the  rough  draft  instructional  materials  under  the  guidance  of  the 
contractor's  instructional  psychologists  (IP's).  Accordingly,  it  was 
determined  that  the  squadron  would  establish  a  Training  Development  Department 
(TDD)  composed  of  E-2C  pilots  and  NFO's,  and  that  the  contractor  would  establish 
a  field  office  co-located  with  the  squadron  and  staffed  with  IP's  and  support 
personnel.  Portions  of  the  effort,  including  graphic-arts  services,  were 
allocated  to  the  contractor's  main  offices,  however.  To  accomodate  the 
development  of  a  corresponding  ISD-based  training  program  for  E-2B  pilot/copilot 
training,  it  was  further  determined  that  RVAW-110,  NAS  Miramar,  would  direct  the 
efforts  of  its  ISD  department  to  the  authoring  of  E-2B  instructional  materials 
through  conversions  of  the  Phase  I  E-2C  analyses  and  the  Phase  II  E-2C  draft 
lessons  under  the  direction  of  the  contractor.  The  contract  was  subsequently 
awarded  on  5  October  1976.  Kick-off  meetings  were  held  at  RVAW-120  on 
26  October  and  RVAW-110  on  3  November.  At  this  time,  the  contractor's  work 
plan  was  presented  and  the  various  responsibilities  were  delineated.  The 
proposed  Work  Schedule  is  shown  in  Figure  1.  The  tasks  of  Instructional 
Materials  Development  and  ATSS  Data  Encoding  were  assigned  to  the  Navy,  with 
the  contractor  in  a  supervisory/support  role.  The  tasks  of  Evaluation  Planning, 
Implementation  Planning,  and  Instructor  Training  Course  Development  were 
assigned  to  the  contractor  with  the  Navy  in  a  consulting  role. 
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The  program  commenced  on  schedule.  At  the  half-way  point  in  the  In¬ 
structional  Materials  Development  schedule,  however,  two  modifications  to 
the  basic  contract  became  necessary.  The  first  involved  the  transfer  of  the 
ATSS  Data  Encoding  task  to  the  contractor,  and  the  second  involved  the  hiring 
of  two  contractor  SME's  to  supplement  the  h'avv's  manpower  commitment  to  the 
Instructional  Materials  Development  task.  The  addition  of  these  resources 
greatly  facilitated  the  completion  of  the  developmental  tasks  and  the  im¬ 
plementation  of  the  program  within  the  originally  specified  contract  period. 

A  second  contract  was  awarded  prior  to  the  completion  of  the  Phase  II 
tasks  to  provide  contractor  support  for  a  follow-on  revision  period.  Figure 
2  delineates  the  tasks  involved  in  that  effort  and  depicts  the  proposed  Work 
Schedule.  The  primary  tasks  were  an  extension  of  the  summative  evaluation,  the 
development  of  further  revision  specifications,  and  the  production  of  revised 
training  materials.  This  effort  was  based  on  a  projected  requirement  to  revise 
approximately  20%  of  the  total  instructional  materials  and  to  "fine-tune"  the 
system  as  a  result  of  input  from  the  summative  evaluation.  The  20%  revision 
requirement  was  later  increased  to  accomodate  a  major  change  in  the  NFO  mission 
software,  and  the  contract  was  amended  accordingly. 

Secondary  tasks  in  the  follow-on  contract  involved  the  development  of  a 
comprehensive  individualized  Instructor-Under-Training  (IUT)  Syllabus.  Of 
particular  importance  to  that  effort  was  the  inclusion  of  a  series  of  slide/tape 
presentations  for  generic  use  in  training  the  ISD  prescribed  procedures  for 
development,  implementation,  and  quality  control. 

PROBLEM/ BACKGROUND 

The  E-2  project  has  been  an  effort  to  apply  the  principles  of  ISD  to 
operator  training  in  an  existing  system.  The  training  requirement  involves 
five  aircrew  positions;  the  pilot,  copilot,  combat  information  center  officer 
(CICO),  air  control  officer  (ACO),  and  the  flight  technician  (FT).  This 
carrier-based  aircrew  must  be  trained  to  perform  the  primary  mission  of  Air¬ 
borne  Early  Warning  (AEW)  and  the  secondary  missions  of  Surface-Subsurface- 
Surveillance  Control  (SSSC),  Anti-Submarine  Warfare  (ASW)  Support,  Search 
and  Rescue  (SAR),  Lost  Plane/Emergency  Tanking,  E-2  Controlled  Approach, 

Anti-PT  Boat  Tactics,  and  Command  and  Control.  Training  for  the  E-2C  is 
conducted  at  RVAW-120  NAS,  Norfolk.  Training  for  the  E-2B  is  conducted  at 
RVAW-110  NAS,  Miramar.  During  the  Phase  I  analysis  and  in  the  development 
period  of  the  Phase  II  effort,  training  at  RVAW-120  utilized  classroom  pre¬ 
sentations  for  academic  training,  the  15F8  Tactics  Trainer  and  the  E-2C 
aircraft  for  NFO/FT  performance  training,  and  the  Cockpit  Procedures  Trainer 
(CPT)  and  the  E-2C  aircraft  for  pilot/copilot  performance  training.  At  this 
time,  an  Operational  Flight  Trainer  (OFT)  was  under  procurement.  Training  for 
E-2B  pilots  at  RVAW-110  utilized  a  similar  program  with  a  more  sophisticated 
E-2B  CPT  and  the  E-2B  aircraft.  Procurement  of  an  E-2C  CPT  and  an  E-2C  OFT 
was  also  in  progress  for  E-2B/E-2C  training  at  RVAW-110. 
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Upon  completion  of  the  Phase  I  ISD  analyses,  the  contractor  determined 
that  the  existing  E-2  Training  programs  could  be  enhanced  in  both  learning 
effectiveness  and  cost-effectiveness  through  conversion  to  an  individualized 
instructional  system  developed  within  the  context  of  a  Systems  Approach  to 
Training.  The  proposed  instructional  system  provided  for  a  series  of  modules, 
most  often  beginning  with  cognitive  lessons  to  present  academic  prerequisites 
and  ending  with  performance  applications  in  practice  or  sortie/scenario 
lessons.  Thus  a  building-block  approach  was  designed  within  and  among  modules 
leading  to  the  final  course  criterion  of  NATOPS  qualification.  Each  Lesson 
Specification  was  based  on  Behavioral  Objectives  derived  from  a  Task  Analysis. 
Provisions  were  made  for  criterion  testing  of  all  Training  Objectives  covered 
in  each  lesson.  Tape/slide  presentations  were  selected  for  the  primary 
cognitive  instructional  media,  with  self-scoring  answer  sheets  serving  the 
testing  function.  Existing  simulators  and  those  under  procurement  were  then 
integrated  into  the  syllabus  along  with  the  cognitive  lessons  and  E-2  air¬ 
craft  flights  in  a  manner  directly  traceable  to  the  Behavioral  Objectives. 
Finally,  the  addition  of  a  tape/slide  presentation  capability  in  the  CPT 
was  recommended  to  increase  its  utility  in  pilot/copilot  training  within 
the  ISD-based  system. 

The  individualized  instructional  system  which  was  specified  for  the  E-2 
aircrew  was  designed  to: 

promote  learning  efficiencies  through  self-pacing,  immediate 
feedback,  early  hands-on  training,  and  criterion  referencing; 

increase  instructor  effectiveness  through  direct  student 
contact  (one-to-one)  as  needed;  and 

enhance  training  system  operations  through  flexible  scheduling. 


(1) 

(2) 

(3) 
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SECTION  II 


INSTRUCTIONAL  SYSTEMS  DEVELOPMENT 


BACKGROUND 

Much  of  the  Instructional  System  Development  (ISD)  methodology  has  been 
in  existence  for  many  years.  However,  it  is  only  within  the  last  few  years 
that  the  methodology  has  been  formalized  into  a  usable,  documented  technology. 
According  to  a  definition  given  in  Air  Force  Manual  50-2,  Instructional  System 
Development  (1970),  ISD  is  a: 

"deliberate  and  orderly  process  for  planning  and  developing 
instructional  programs  which  insures  that  personnel  are  taught 
the  knowledges,  skills,  and  attitudes  essential  for  successful 
job  performance." 

This  is  obviously  not  a  new  goal  in  aircrew  training  or  educational  technology, 
as  a  whole.  The  new  aspect  that  ISD  brings  to  training  is  the  use  of  systems 
analysis  techniques  in  an  attempt  to  relate  the  training  objectives  to  realistic 
job  performance  requirements.  That  is,  teach  the  individual  all  of  the 
information  which  is  necessary. 

Through  the  conduct  of  the  present  E-2  program  and  a  previous  program  on 
a  very  different  multicrew  aircraft  system  (Air  Force  B-l  strategic  bomber), 
the  contractor  examined,  evaluated,  and  expanded  the  Systems  Approach  to 
Training,  SAT  methodology.  The  total  "system"  of  information  involved  in  the  SAT 
methodology,  as  defined  by  the  contractor,  is  illustrated  in  Figure  3.  This  sys¬ 
tematic  approach  to  information  accumulation  and  application  ensures  that  the 
analysis  is  comprehensive  and  logical.  The  conceptualized  flow  of  information  high¬ 
lights  numerous  interactions  among  facts,  assertions,  assumptions,  and  constraints. 
Some  attempt  was  made  to  order  (vertically)  the  blocks  in  a  chronological  manner 
according  to  approximately  when  they  take  predominance  in  consideration.  The 
blocks  with  the  same  number  share  some  common  attribute.  A  brief  description 
follows. 

The  focus  of  the  SAT  methodology  is  centered  around  making  the  training 
responsive  to  the  mission  requirements  of  the  man-machine  system.  The  man- 
machine  system  can  be  partitioned  into  two  components,  functions  performed  by 
the  human  operator  (la)  and  functions  performed  by  the  machine  (lb).  These 
functions  and  the  interface  between  the  man  and  the  machine  are  inputs  to  the 
training  analysis.  In  addition,  there  are  often  operational  policies  (lc) 
that  impact  upon  the  training  system  which  must  be  addressed.  Although  these 
policies  are  input  to  the  system  initially,  it  is  possible  that  the  training 
analysis  reveals  information  that  subsequently  leads  to  policy  changes. 

Another  set  of  inputs  to  the  training  analysis  involves  the  economic  information 
(Id)  pertaining  to  such  things  as  the  costs  of  instructional  material,  training 
devices,  facilities,  and  personnel. 
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ANALYSIS 


Figure  3.  System  Components  for  the  Development  of  Training 
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The  first  phase  of  the  analyses  includes  determining  and  documenting 
the  human  operator  functions  in  terms  of  a  task  analysis  (2a).  The  machine 
functions  are  also  analyzed  to  establish  the  external  influences  (or  con¬ 
straints)  (2b)  that  are  imposed  on  the  training  program  (e.g.  ,  airborne 
safety  considerations).  Another  contributor  to  the  external  influences 
includes  the  policies  of  both  the  training  and  the  operational  communities. 
The  trainees'  incoming  skills  and  knowledges  (2c)  are  a  function  of  the 
operational  policy  in  terms  of  the  determination  of  the  sources  of  trainees. 
The  external  influences  and  Incoming  skills  and  knowledges  can  be  considered 
as  "boundary  conditions"  for  the  training  analysis.  Those  behaviors  which 
must  be  demonstrated  by  the  trainees  before  graduation  are  logically 
grouped  into  behavioral  objectives  (3)  along  with  synopses  of  training¬ 
relevant  data  pertaining  to  each  objective. 

The  greatest  point  of  departure  of  this  SAT  "model"  from  traditional 
flow  charts  is  the  iterative  process  indicated  by  the  "cogwheel”  supporting 
blocks  4a,  b,  and  c.  The  training  objectives  (4a)  may,  in  a  simplified 
way,  be  thought  of  as  those  behavioral  objectives  not  already  in  the  in¬ 
coming  repertoire  of  behavior  (but  couched  in  training  terms  rather  than 
operational  mission  terms).  Methods  (instructional  strategies)  and  media 
(training  devices)  are  specified  (4b)  which  can  supply  the  appropriate 
training  environment.  Instructional  sequencing  of  objectives  into  courses 
(4c)  provides  a  syllabus.  Each  of  these  blocks  (4a,  b,  c)  are  interde¬ 
pendent  in  that  multi-dimensional  criteria  are  applied.  For  example,  the 
selection  of  a  training  device  is  dependent  on  economic  factors,  including 
device  utilization.  Device  utilization,  on  the  other  hand,  interacts  with 
the  training  objectives  via  the  level  determined  for  incoming  skills  and 
knowledges.  But  still,  Che  training  objectives  themselves  are  formulated 
with  some  knowledge  of  the  devices  that  may  be  available  for  Instructing  and 
testing.  To  aid  in  this  process  of  developing  the  syllabus,  a  model  of  the 
instructional  system  (4d)  may  be  invaluable,  depending  on  the  degrees  of 
freedom  allowed  to  the  training  analysts.  The  model  takes  as  inputs  the 
hypothesized  syllabus  (4c),  the  external  influences  (2b),  the  incoming 
skills  and  knowledges  (2c),  and  the  economic  data  (Id),  to  determine  the 
ramifications  on  the  training  system  (5b)  via  critical  descriptors.  The 
detailed  description  of  the  syllabus  itself  is  contained  in  the  lesson 
specifications  and  performance  measures  (5a).  Implementation  (6a)  of  the 
latter  provides  evaluation  data  (7)  for  quality  control  feedback,  while 
the  training  system  descriptors  provide  planning  data  (6b)  for  major 
alterations  in  the  structure  of  the  overall  instructional  system. 

The  SAT  methodology,  as  discussed  here,  is  not  an  algorithm  that  "spits 
out"  answers;  but  rather.  It  is  a  conceptual  framework  that  serves  as  a 
decision  aid  to  be  used  in  documenting  and  presenting  the  information  to 
decision  makers.  The  relative  emphasis  on  the  various  components  of  the 
SAT  process  is  different  depending  upon  whether  one  is  developing  a  future 
training  program,  modifying  an  existing  program,  or  investigating  a 
particular  component  of  the  system. 
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ASSUMPTIONS  (GENERIC) 

The  following  assumptions  apply  to  the  contractor's  approach  to  ISD 
for  operator  training  in  an  existing  system: 

*  Principles  of  learning  and  a  technology  of  instruction  are 
now  available  which  can  be  successfully  applied  in  the 
operational  training  setting. 

*  The  application  of  ISD  to  an  existing  training  system  will 
result  in  significant  increases  in  both  learning  effectiveness 
and  cost  effectiveness,  while  providing  greater  responsiveness 
to  the  needs  of  the  operational  environment. 

*  The  most  effective  approach  to  ISD  is  not  a  fully  proceduralized 
one,  but  rather  a  systematic  one  following  the  traditional  steps, 
while  recognizing  the  iterative  nature  of  the  processes  involved 
and  relying  on  expert  judgments. 

ASSUMPTIONS  ( E— 2  SPECIFIC) 

The  entire  Instructional  System  Development  (ISD)  process  previously 
described  consists  of  decision  processes  based  on  data.  The  data  are  com¬ 
posed  of  descriptive  facts,  research  results,  assertions  (the  result  of 
prior  decisions),  assumptions,  and  constraints.  This  section  will  enumerate 
those  items  which  fall  into  the  category  of  assumptions.  Although  these 
"external  influences"  are  subject  to  change  (and  in  fact,  have  changed),  the 
following  are  those  that  either  were  agreed  upon  by  NAVTRAEQUIPCEN 'S  contract 
monitor  to  be  a  reasonable  basis  for  accomplishing  the  work  under  the  con¬ 
tract  or  which  were  implied  or  stated  in  the  statement  of  work. 

*  The  proposed  training  program  will  be  for  all  crew  stations  of  the 
E-2C  aircraft  and  will  be  implemented  by  personnel  of  the  RVAW-120 
Squadron,  Norfolk  Naval  Air  Station.  (This  assumption  applied  to 
Phase  I  only.  The  scope  of  the  effort  was  expanded  to  include  the 
E-2B  pilot/copilot  training  task  prior  to  award  of  the  Phase  II 
contract. ) 

*  The  trainee  flow  will  remain  constant  at  approximately  30  NFO/FT  and 
36  pilots  per  year  and  will  derive  from  the  same  sources  including 
both  "nuggets,"  Undergraduate  Pilot  Training/Undergraduate  Navigators 
Training  (UPT/UNT)  graduates,  and  transitioning  crews.  In  particular, 
it  was  assumed  that  the  West  Coast  E-2C  pilots  will  not  be  processed 
through  RVAW-120.  (This  assumption  was  modified  after  the  start 

of  Phase  II.  A  separate  track  of  the  pilot  syllabus,  using  portions 
of  the  primary  ISD-based  syllabus,  was  developed  to  accommodate  this 
change. ) 

*  The  Airborne  Intercept  Control  (AIC)  course  will  remain  at  Dam  Neck, 
i.e.,  it  will  not  become  part  of  the  RVAW-120  curriculum.  (This 
assumption  was  modified  after  the  start  of  Phase  II.  An 

existing  non-ISD  curriculum  was  incorporated  into  the  overall  ISD- 
based  system  to  accommodate  this  change.) 
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*  No  major  equipment  (simulators,  trainers)  will  be  procured  to 
support  the  proposed  training  program,  but  the  Operational 
Flight  Trainer  (OFT)  will  be  ready  for  training  at  the  time 
of  Implementation.  (This  assumption  was  also  modified.  An 
interim  syllabus  was  developed  during  the  final  months  of  the 
Phase  II  Instructional  Materials  Development  effort  to  accomodate 
this  change. ) 

*  Time  available  on  existing  training  equipment  will  not  be  less 
than  currently  available;  simulator  time  will  be  proportioned 
equally  between  use  by  trainees,  operational  crew  members,  and 
scheduled  maintenance. 

*  No  major  hardware  or  software  changes  will  impact  on  E-2  training 
during  Phase  II  of  the  ISD  efforts.  (This  assumption  was  modified. 
A  separate  track  within  the  NFO/FT  syllabus  was  developed  to 
accomodate  a  major  software  change  for  the  radar  processing 
system.  The  incorporation  of  the  ”03"  software  required  a 

major  syllabus  revision  approximately  one  year  after  implementa¬ 
tion). 

*  Navy  SME's  can  be  effectively  utilized  in  the  authoring  of 
instructional  materials  (under  the  direction  of  the  con¬ 
tractor's  IP's)  as  well  as  in  the  tasks  of  Implementation, 
Evaluation,  and  Revision. 

*  Sufficient  SME  time  will  be  provided  on  a  non-interference 
basis  by  RVAW-120  and  RVAW-110  to  accommodate  the  Phase  II 
production  schedule.  (This  assumption  changed  requiring  a 
contract  modification  to  provide  two  contractor 

SME's  aa  a  supplement  to  squadron  ISD  resource  personnel. 
Additionally,  support  was  provided  from  the  Wing  on  a 
temporary  duty  basis.) 

*  An  Educational  Specialist  will  be  provided  to  assist  in  the 
ATSS  Data  Encoding  task.  (The  loss  of  this  billet  necessitated 
a  contract  change  to  provide  further  contractor  personnel 
support. ) 

*  Existing  slides  for  E-2  training  support  can  be  successfully 
and  cost-effectively  integrated  into  audio-visual  lessons 
developed  within  the  ISD  framework. 

*  Adaptation  of  the  E-2C  ISD  program  to  the  additional  require¬ 
ment  of  E-2B  pilot  training  can  be  accommodated  with  a  level  of 
effort  and  a  resource  expenditure  of  approximately  50%  of 

that  required  for  an  original  development  effort. 
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The  contractor  responded  to  the  problem  of  developing  an  E-2  SAT 
program  by  coordinating  the  skills  and  experiences  of  specialists  in  the 
disciplines  of  psychology,  education,  human  factors  engineering,  and 
aviation.  Each  of  these  specialists  was  selected  for  the  effort  on  the 
basis  of  prior  operational  and/or  training  experience  with  military  air¬ 
craft.  In  addition,  one  contractor  analyst  (a  pilot/navigator  -  rated 
engineer)  received  additional  subject  matter  expertise  through  partici¬ 
pation  in  an  E-2C  NFO  transition  course  at  RVAW-120.  These  contractor 
personnel  then  performed  the  analysis  and  design  effort  in  accordance 
with  the  contractor's  "System  Components  for  the  Development  of  Training" 
model  and  associated  procedures  developed  in  a  SAT  effort  for  the  B-l 
aircrew.  The  Phase  I  effort  was  subsequently  documented  in  NAVTRAEQUIPCEN 
Report  No.  75-C-010L-1.  A  brief  description  of  the  approach  and  subsequent 
design  recommendations  is  given  in  the  paragraphs  below. 

The  development  of  a  comprehensive  data  base  was  the  first  task  in 
applying  the  contractor's  SAT  model  to  the  E-2C  training  program.  This 
was  accomplished  through  a  two-faceted  analysis.  One  facet  involved  an 
extensive  analysis  of  the  existing  training  program.  The  second  facet 
consisted  of  a  task  analysis.  The  background  information  for  the  NFO 
task  analysis  was  established  through  firsthand  observation  and  recording 
of  classroom  presentations  conducted  by  the  RVAW-120  squadron  training 
personnel.  While  this  appeared  to  be  the  most  efficient  in  terms  of  ob¬ 
taining  information  about  the  NFO/FT  training,  the  contractor's  previous 
experience  with  the  pilot/co-pilot  training  task  and  the  availability  of 
documentation  (including  recently  developed  audio-visual  presentations) 
for  a  significant  portion  of  that  training,  alleviated  the  necessity  for 
a  corresponding  approach.  In  both  cases,  however,  all  available  training 
documentation  was  collected  for  analysis.  The  following  are  representative 

a.  Instructor  Guides  (in-house) 

b.  Student  Handouts 

c.  Class  Lecture  Notes 

d.  CARAEWTRARON  120  INSTRUCTION  1500. 1A 

e.  NATOPS  (NAVAIR  01-E-2AAA-1) 

f.  OPERATIONAL  DIAGNOSTIC  FLIGHT  MANUAL  (NAVAIR  01-E2AAA-1A. 2) 

g.  OPERATIONAL  SOFTWARE  FLIGHT  MANUAL  (NAVAIR  0I-E2AAA-1A. 1 ) 

h.  CREW  OPERATORS  CHECKLISTS  (NAVAIR  01-E2AAA-1C) 

i.  CHECKLIST-POCKET  (NAVAIR  01-E2AAA-1B) 

j.  GRUMMAN  Instructor  Program  Guides 

k.  Operator  Manuals  for  Training  Devices 

l.  EDUCATIONAL  SELF-AUDIT  (RVAW-120,  CNETS) 

m.  NAVAL  FLIGHT  OFFICER  FUNCTION  ANALYSIS:  E-2B  CICO  (NAMRL) 
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These  informational  sources  were  then  supplemented  by  informal  inter¬ 
views  with  the  RVAW-120  training  personnel  and  direct  observations  of 
operational  equipment/simulators  and  crews.  In  this  procedure,  heavy 
reliance  was  made  on  two  sources:  subject  matter  expertise  and  operational 
documentation.  These  sources,  together  with  classroom  lecture  notes 
covering  operational  information  not  otnerwise  available,  provided  the  con¬ 
tractor  with  the  basic  information  for  preparing  the  task  analysis  listing. 

The  format  which  was  employed  for  the  task  analysis  was  one  adopted 
from  experience  with  a  previous  training  program  design.  The  resulting 
data  base  consisted  of  a  detailed  hierarchical  nreakdown  of  all  activities 
required  to  perform  the  mission.  Of  special  significance  was  the  level 
of  detail  (task  element  level),  the  mission-performance/system-operation 
orientation,  and  the  display-operator-control  relationships.  Further¬ 
more,  an  additional  effort  was  made  to  include  remarks  pertinent  to  task 
performance  but  not  directly  reflected  in  the  action  statement.  Much 
attention  was  given  to  this  effort,  as  it  was  designed  to  provide  the 
information  necessary  to  formulate  behavioral  objectives  and  to  provide 
Lne  structure  of  the  lesson  specifications  later  in  the  program.  The 
task  analysis  was  sent  to  RVAW-120  and  to  NAVTRAEQU1PCEN  for  review. 

Input  from  RVAW-120  squadron  training  personnel  was  used  in  revisions  of 
the  task  analytic  information  as  it  was  incorporated  into  the  lesson 
specifications. 

The  next  step  involved  the  identification  of  training  objectives. 

This  was  basically  a  two-stage  process.  First,  the  task  analysis  was 
partitioned  into  a  listing  of  discrete  behavioral  components  that 
formulate  the  behavioral  objectives.  Included  in  the  objectives  are 
key  features  such  as  ratings  of  performance  limits.  As  will  be  seen  later  in 
the  discussion  of  lesson  specification  development,  the  behavioral 
objectives  play  a  key  role  in  establiooing  performance  assessment 
criteria.  The  behavioral  objectives,  the  second  deliverable,  were  sent 
to  NAVTRAEQUIPCEN  and  RVAW-120  for  verification.  The  second  stage  in¬ 
volved  the  "subtraction"  of  the  incoming  skills  and  knowledges.  These 
incoming  skills  and  knowledges  for  various  trainees  were  established 
through  discussions  with  RVAW-120  personnel  as  well  as  an  analysis  of 
documents  related  to  their  previous  training.  Thus,  the  behavioral 
objectives  minus  the  incoming  skills  and  knowledges  resulted  in  the 
listing  of  actual  training  objectives.  These,  too,  were  later  used  in 
the  preparation  of  lesson  specifications  as  the  key  determinants  of 
content  areas. 

Integral  to  the  preparation  of  lesson  specifications  were  the  media 
analysis  and  the  determination  of  instructional  strategies  and  sequences. 

The  specification  of  training  devices  was  based  on  the  training  objectives 
identified  during  the  SAT  process  and  was  oriented  toward  obtaining  the 
best  possible  trainee  performance  at  the  least  possible  cost.  The 
sequencing  of  instructional  blocks  was  based  on  both  learning  principles 
and  effective  resource  (device)  utilization. 
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The  third  and  final  task,  resulting  in  a  deliverable,  was  the 
preparation  of  lesson  specifications.  As  alluded  to  earlier,  each  of 
the  previous  steps  contributed  directly  to  the  product  of  this  effort. 

The  behavioral  objectives,  operational  policy,  availability  of  resources, 
and  incoming  skills  and  knowledges  played  a  key  role  in  the  instructional 
design.  Upon  this  information,  the  decision  was  made  to  develop  an 
individualized  instructional  system  using  audio-visual  presentations. 
Analysis  of  existing  training  devices/simulators  and  an  additional  oper¬ 
ational  flight  trainer  under  procurement  indicated  that  no  costly  hardware 
purchases  would  be  necessary.  It  did  appear,  however,  that  training 
effectiveness  could  be  enhanced  by  a  more  optimum  use  of  these  devices. 

The  procurement  of  audio-visual  carrels  and  a  modification  of  the  Cockpit 
Porcedures  Trainer  (addition  of  audio-visual  capability)  were  the  only  new 
hardware  requirements  indicated. 

The  instructional  strategy  was  developed  in  consideration  of  the 
student  flow,  the  incoming  skills  and  knowledges,  available  training 
resources,  training  objectives,  and  the  concept  of  early  hands-on 
training.  The  format  of  the  lesson  specification  was  thus  designed  to 
reflect  an  individualized  approach,  with  instruction  divided  into  three 
continuous  domains  -  the  cognitive,  practice,  and  sortie/scenario.  In 
the  cognitive  domain,  the  sequence  of  instruction  was  designed  to  pre¬ 
sent  in  a  hierarchical  manner  (building  a  network  of  associations  upon 
the  developing  repertoire)  that  information  which  directly  contributes 
to  mission  performance  and/or  system  operation.  Accordingly,  the  struc¬ 
ture  of  the  training  objectives  was  used  as  the  basis  for  the  teaching 
points  in  the  lesson  specifications.  The  ordering  of  the  -objectives  was 
permuted,  however,  in  consonance  with  learning  strategies. 

To  facilitate  the  media  lessonware  preparation,  the  cognitive  in¬ 
struction  was  divided  into  teaching  points  with  corresponding 
descriptions  of  media  support.  An  effort  was  made  to  segment  the  teaching 
points  into  the  smallest  units  of  information  appropriate  for  evaluation. 

In  many  cases,  however,  a  number  of  tape/slide  combinations  were 
envisioned  for  the  support  of  a  single  teaching  point. 

Cover  sheets  were  developed  which  identify  the  lesson  name  and  number, 
training  objectives  covered,  evaluation  criteria,  prerequisite  lessons, 
training  devices  required,  projected  training  time,  and  any  additional 
references.  The  cover  sheets  for  cognitive  lessons  were  followed  by  the 
sequence  of  instruction.  Likewise,  cover  sheets  for  practice  and  sortie/ 
scenario  lessons  were  followed  by  behavioral  objectives  containing 
step-by-step  procedures,  conditions,  and  standards. 

METHODOLOGY  (PHASE  II) 

Based  on  the  analyses  and  subsequent  design  recommendations  which 
emerged  from  the  Phase  I  E-2C  ISO  effort,  the  contractor  undertook  the 
tasks  of  Instructional  Materials  Development,  ATSS  Data  Base  Development, 
Evaluation  Planning,  Implementation  Planning,  Instructor  Training  Course 
Development,  Implementation/Evaluation,  and  Revision  Consultation  in 
Phase  II.  The  model  which  provided  the  direction  for  Phase  I  continued 
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Co  be  used  throughout  Phase  II,  thus  ensuring  a  comprehensive,  iterative  ' 

approach.  However,  a  major  difference  in  the  employment  of  this  approach 
was  incorporated  into  the  Phase  II  methodology.  This  difference  involved 
a  greater  reliance  on  Navy  SME's,  using  them  for  the  authoring  of  rough 

draft  instructional  materials  rather  than  as  consultants  on  an  input-  j 

output  basis.  While  details  of  the  implementation  are  presented  in 

Section  III  of  this  report,  the  basic  methodology  is  discussed  in  a  j 

step-by-step  manner  in  the  following  paragraphs. 

The  first  task  within  the  Phase  II  methodology,  instructional  Materials 
Development,  was  also  the  most  complex,  pervasive,  and  resource-consuming 
task.  Primary  inputs  were  the  instructional  system  design,  the  training 
objectives,  the  syllabus  flow  charts,  and  the  lesson  specifications.  This 
task,  which  spanned  a  period  of  twenty  months,  was  comprised  of  the  follow¬ 
ing  sub-tasks:  lesson  specification  review;  detailing  of  lesson  specifications; 
authoring  and  test  development;  production;  and  materials  tryout  and  formative 
revision.  The  overall  effort  was  managed  by  the  contractor's  principal  in¬ 
vestigator,  the  TDO  (and  training  model  manager)  from  RVAW-120,  and  the  ISD 
officer  from  RVAW-110.  SME's  were  integrated  into  the  effort  through  the 
contractor's  author-training  materials  and  on-the-job  training  (OJT). 

Lesson  Specification  Review  preceded  all  other  sub-tasks  and  involved  a 
general  course  review  as  well  as  a  pre-authoring  activity  of  the  SMEs.  The 
course  review  process  was  conducted  primarily  by  management  personnel  (IPs  and 
senior  SMEs)  utilizing  the  syllabus  flow  charts.  At  this  level,  SME  input 
resulted  in  some  restructuring  of  the  syllabus,  with  additions,  permutations, 
and  combinations  of  proposed  lesson  materials  when  required.  All  proposed 
revisions  were  carefully  documented  and  linked  to  the  task  analysis  and  be¬ 
havioral  objectives  documentation.  The  most  significant  revisions  emerged 
from  a  need  for  more  emphasis  on  the  tactical  areas,  changes  in  the  hardware/ 
software,  and  squadron  constraints  regulating  trainer  and  flight  time.  This 
effort  was  most  heavily  concentrated  in  the  first  months  of  the  contract,  but 
continued  at  a  lower  level  throughout  the  entire  instructional  materials  develop¬ 
ment  task.  The  pre-authoring  activity  of  the  SMEs  within  this  sub-task  was  at  the 
more  specific  intra-lesson  level,  taking  inputs  from  the  lesson  specifications 
developed  by  the  contractor  in  Phase  I.  The  effort  was  primarily  directed  toward 
revisions  of  the  learning  objectives  which  identified  the  proposed  scope  of  the 
lessons,  but  extended  to  the  level  of  supporting  teaching  points  when  required. 

The  performance  period  extended  throughout  the  authoring  sub-task  and  in  con¬ 
junction  with  that  process. 

Detailing  of  Lesson  Specifications  was  a  sub-task  which  was  primarily 
the  function  of  Navy  and  contractor  SMEs.  Here,  the  general  outlines  of  the 
lesson  specifications  were  expanded  with  the  technical  information  necessary 
to  provide  specific  teaching  points.  The  output  of  this  sub-task  was  a  set 
of  detailed  lesson  specifications  to  serve  as  guides  for  storyboarding  tape/ 
slide  lessons  and  authoring  classroom,  practice,  and  sortie/scenario  lessons. 

This  documentation  further  served  as  input  to  the  ATSS  data  base  and  as  student 
handouts  to  augment  the  mediated  lessons. 
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Authoring  and  Test  Development,  the  most  time-consuming  sub-task,  followed 
the  detailing  of  lesson  specifications  and  was  primarily  carried  out  by  Navy 
and  contractor  SMEs ,  with  editing  by  contractor  technical  writers  and  IPs. 

Three  basic  formats  were  employed.  For  classroom  presentations,  an  existing 
Navy  lecture  guide  format  was  adapted.  The  SMEs  then  transformed  the  detailed 
lesson  specifications  into  this  format,  developed  corresponding  test  items 
(based  on  the  training  objectives),  and  specified  supporting  overhead  transpar¬ 
encies  when  required.  For  practice  and  sortie/scenario  lessons,  a  new  format 
was  developed  which  presented  proceduralized  instructions  for  the  students  and 
the  instructors  and  incorporated  criterion-referenced  testing.  For  tape/slide 
presentations,  a  detailed  process  of  "storyboarding'’  was  employed.  The  format 
provided  for  a  frame  by  frame  description  of  the  visual,  a  corresponding  narra¬ 
tive,  and  related  test  items.  The  output  of  the  storyboarding  effort  consisted 
of  several  thousand  pages  of  working  documentation  for  input  to  the  audio  and 
visual  production  processes.  A  flow-chart  depicting  the  corresponding  review 
process  is  presented  in  Figure  4.  The  most  significant  problems  at  the 
"storyboard"  level  arose  from  deficiencies  in  the  source  documentation 
(e.g.  NATOPS),  lack  of  detail  in  production  notes,  and  a  tendency  of  the  SMEs 
to  put  too  much  into  one  frame  of  instruction  or  to  assume  too  much  pre¬ 
requisite  learning.  The  extensive  review  process  focused  on  reducing  these 
discrepancies  and  ambiguities.  Production  then  followed  as  a  contractor  sub¬ 
task,  with  typing,  printing,  and  duplication  support  for  "hard  copy"  materials, 
and  the  complex  audio-visual  processes  described  in  Appendix  A  for  tape/slide 
lesson  materials. 

The  second  task,  and  one  which  proceeded  concurrently  with  Instructional 
Materials  Development,  was  the  ATSS  Data  Base  Development.  Inputs  for  this 
task  were  the  source  documentation  and  all  the  analytical  documentation  from 
the  Phase  I  design,  and  the  actual  instructional  materials  from  Phase  II. 

The  task  required  a  joint  effort  between  the  contractor  and  the  Naval  Weapons 
Center,  China  Lake.  A  conceptualization  of  the  Instructional  Systems  Development 
Impact  System  Directory  (ISDISD)  was  first  formulated,  followed  by  the  specifi¬ 
cation  of  detailed  data  formats.  The  purpose  of  the  ISDISD  was  to  provide 
linkages  between  all  data  bases  of  ISD  documentation  to  facilitate  future 
revisions  to  the  curriculum.  The  contractor's  effort  was  to  encode  the  data, 
while  the  Naval  Weapons  Center's  effort  was  to  keypunch  the  data  and  to  develop 
the  corresponding  computer  programs  in  accordance  with  NTEC's  specification 
(MIL-T-29053) .  The  outputs  of  this  task  were  the  following  data  bases  for  the 
E-2C  documentation,  with  corresponding  linkages:  the  Task  Analysis  Data  Base, 
the  Behavioral  Objectives  Data  Base,  the  Hardware  Data  Base,  the  Software  Data 
Base,  and  the  Lesson  Data  Base. 

The  third  task,  Evaluation  Planning,  was  conducted  by  the  contractor's 
IP's,  with  input  and  review  by  RVAW-120  and  RVAW-110  SME's  and  management 
personnel.  Inputs  for  this  task  were  the  instructional  system  design,  the 
Training  Objectives,  the  course  syllabus,  and  squadron  resources  and  con¬ 
straints.  This  two-month  effort  was  conducted  early  in  the  program,  provid¬ 
ing  for  evaluation  of  both  the  aircrew  training  system  and  the  Instructor 
Training  Course  (ITC).  The  contractor's  IP's  first  developed  a  compre¬ 
hensive  model  unique  to  the  requirements  of  the  E-2  ISD-based  training 
system,  ind  then  specified  procedures  and  formats  for  data  collection.  The 
resulting  Evaluation  Plan  addressed  both  formative  and  summative  evaluation 
requirements  and  procedures.  Provisions  were  made  to  collect  data  for  both 
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internal  and  external  validation.  Additionally,  a  conceptualization  for 
incorporating  evaluation  data  into  the  ATSS  was  formulated.  The  Evaluation 
Plan  further  specified  the  format  and  procedures  for  preparing  Revision 
Specifications. 

The  fourth  task  in  the  contractor's  methodology  involved  Implementation 
Planning.  This  two-month  effort  was  performed  after  Evaluation  Planning 
and  Instructor  Training  Course  Development,  using  inputs  frcn  both.  Major 
inputs  were  also  derived  from  the  instructional  systems  design,  the  syllabus 
flow  charts,  and  the  course  materials.  Squadron  resources  and  constraints 
were  particularly  important  considerations  in  this  task.  The  effort  was 
conducted  primarily  by  the  contractor's  IP's,  with  squadron  management  in  a 
significant  consulting  role.  The  resulting  Implementation  Plan  provided  a 
detailed  description  of  the  training  system,  the  required  resources,  the 
organization  and  functions  of  the  personnel  involved,  and  a  proposed 
implementation  schedule.  Key  features  of  this  plan  were  a  conceptualization 
of  the  Student  Learning  Center  (SLC)  and  guidelines  for  scheduling  student 
flow  through  the  system. 

The  fifth  task  in  the  contractor's  methodology  involved  Instructor 
Training  Course  (ITC)  Development.  This  was  a  four-month  effort  conducted 
by  the  contractor  in  two  phases:  course  development  and  seminar  implementa¬ 
tion.  The  development  effort  was  a  function  of  contractor  IP's,  while 
the  seminar  implementation  involved  contractor  IP's  and  SME's,  Training 
Development  Department  (TDD)  personnel,  and  squadron  instructors.  The 
course  development  utilized  input  from  generic  materials  previously  de¬ 
veloped  by  the  contractor  (and  used  for  SME  training),  and  the  Imple¬ 
mentation  and  Evaluation  Plans.  Key  features  of  the  course  centered 
on  instructor  roles  in  the  conduct  of  training  and  in  the  evaluation/ 
revision  process.  The  course  was  conducted  on  six  different  occasions  to 
allow  all  squadron  personnel  the  opportunity  to  attend.  While  the  ITC  was 
first  implemented  as  a  three-day  seminar  detailing  the  overall  ISD  process, 
subsequent  evaluation  resulted  in  revisions  to  the  presentation  strategy. 
Subsequently,  the  course  was  presented  in  a  one-day  format  focusing  more 
exclusively  on  instructor  roles  and  directing  further  study  of  the  in¬ 
structional  materials. 

The  sixth  task  in  the  contractor's  methodology  consisted  of  Implementation 
and  Evaluation.  All  of  the  previously  described  tasks  contributed  to  this 
effort.  The  contractor's  IP's  and  training  analysts  worked  closely  with  the 
TDD's  and  the  Operations  Departments.  While  delays  in  the  Instructional 
Materials  Developed  effort  impacted  significantly  on  implementation,  the 
effort  was  accommodated  within  the  contract  period  and  all  E-2C  training 
materials  were  available  as  required.  A  phased  implementation  with  some 
individualized  instructional  materials  remaining  temporarily  in  the  classroom 
setting,  was  necessary,  however,  for  E-2B  implementation.  One  additional 
factor,  which  impacted  on  Instructional  Materials  Development  as  well  as 
Implementation,  was  a  delay  in  the  delivery  of  the  E-2C  Operational  Flight  • 
Trainer  (OFT).  An  intermediate  syllabus  relying  more  heavily  on  E-2  flights 
and  the  CPT  was  designed  and  developed  to  accomodate  this  problem.  The  im¬ 
pact  on  Evaluation  was  to  delay  completion  of  summative  data  collection  for 
an  additional  year,  while  formative  data  collection  was  accommodated  within 
the  contract  period. 
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The  seventh  task,  Revision  Consultation,  was  performed  by  the  con¬ 
tractor's  IP's  and  training  analysts  concurrent  with  Implementation. 
Evaluation  data  provided  inputs  for  this  process.  Revision  specif icat ions 
were  prepared  to  assist  in  "fine  tuning"  the  system  and  to  update  the 
training  program  in  accordance  with  external  hardware  and  software  changes. 

A  detailed  revision  documentation  system  was  further  developed  to  facilitate 
the  process. 

The  eighth  and  final  task  in  the  contractor's  methodology  involved 
the  collection  of  Resource  Utilization  Data  for  future  planning  purposes. 
This  effort  spanned  the  entire  contract  period,  taking  inputs  from  Navy 
as  well  as  contractor  resource  expenditures.  Man-hours  and  funds  expended 
by  task  were  documented  along  with  other  relevant  data.  Weekly  time-slips 
were  prepared  by  both  Navy  and  contractor  personnel. 
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SECTION  III 

IMPLEMENTATION 


ORGANIZATION  AND  STAFFING 

A  summary  of  the  overall  Phase  II  project  organization  and  staffing 
is  depicted  in  Figure  5.  General  direction  for  the  project  came  from 
the  NTEC  project  monitor,  who  coordinated  the  efforts  of  RVAW-I20,  RVAW-I10, 
and  the  contractor.  Instructional  Psychologists  from  the  contractor's 
Human  Factors  Section  provided  overall  planning,  project  management,  techni¬ 
cal  direction  and  consultation  support,  as  well  as  the  preparation  of  all 
report  documentation  (e.g.  Evaluation  Plan,  Implementation  Plan,  Progress 
Reports).  The  contractor's  Principal  Investigator  was  located  at  RVAW-120 
NAS  Norfolk.  Assisting  him  at  that  site  were  two  IP's  (who  primarily  con¬ 
tributed  to  the  ATSS,  ITC  and  evaluation  tasks),  an  NFO  SME,  a  technical 
writer/editor,  a  narrator,  and  clerical  personnel.  Off-site  contractor 
support  for  the  project  was  provided  by  a  pilot  training  consultant  (IP) 
who  had  played  a  significant  role  in  the  Phase  I  analysis  and  design  effort, 
an  E-2B  pilot  SME  located  at  RVAW-1I0  NAS  Miramar,  and  media  specialists 
and  graphic  arts  personnel  at  the  contractor's  main  facility  in  Buffalo, 

New  York.  An  IP,  assisted  by  a  technician,  coordinated  the  production  effort. 
A  total  of  7  illustrators,  assisted  by  photographic  services  personnel,  were 
ultimately  involved  in  the  primary  sub-task,  tape/slide  lesson  production. 

SME  support  for  Instructional  Materials  Development  was  provided  by 
RVAW-120  and  RVAW-110.  A  separate  Training  Development  Department  or 
ISD  Department  was  established  at  the  E-2C  and  E-2B  squadrons,  respec¬ 
tively.  The  Training  Development  Officer  (TDO)  for  RVAW-120  and  the 
ISD  officer  for  RVAW-110  provided  a  point  of  contact  for  the  contractor's 
principal  investigator  and  support  personnel,  and  managed  the  efforts  of 
a  team  of  E-2B  and  E-2C  pilot  SMEs  and  E-2C  NFO  SMEs.  Additionally, 

RVAW-120's  TDO  functioned  as  the  training  model  manager.  The  manpower 
level  and  the  composition  of  the  departments  fluctuated  throughout  the 
project,  as  a  result  of  program  demands,  squadron  constraints,  and 
turnover  of  Navy  personnel.  Not  depicted  in  Figure  5,  but  of  equal 
importance  to  the  project,  was  coordination  with  the  squadrons'  Safety 
and  Operations  Departments  for  Instructional  Materials  Development  and 
Implementation.  Man-hours  by  task  for  both  contractor  and  Navy  personnel 
are  presented  in  Section  V  of  this  report. 
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SYLLABUS.  Instructional  materials  were  developed  to  support  three  types 
of  training  for  E-2C  pilots/copilots,  E-2C  NFOs/FTs,  and  E-2B  pilots/co¬ 
pilots:  cognitive  lessons,  practice  lessons,  and  sortie/scenario  lessons. 

The  cognitive  lessons  present  the  ''enabling”  academic  information  via  the 
primary  medium  of  tape/slide  lessons,  the  secondary  medium  of  classroom 
lecture/discussion  lessons,  or  the  supplemental  medium  of  assigned  readings 
in  technical  publications.  The  individualized  tape/slide  lessons  employ  a 
high  degree  of  visual  fidelity,  focusing  on  a  control-display  orientation. 

The  "need-to-know”  systems  information  and  mission  orientation  are  then 
programmed  into  this  context.  Within  the  NFO  curriculum,  this  approach  is 
reflected  in  an  emphasis  toward  operator  interaction  with  the  computer 
software. 

Lesson  specifications  were  adapted  for  use  as  student  handouts  in 
support  of  the  tape/slide  lessons.  For  classroom  lecture/discussion  lessons, 
however,  documentation  was  developed  in  the  form  of  detailed  lesson  guides 
for  both  instructors  and  students.  These  lessons  reflect  a  greater  depth 
in  mission  background  and  the  tactical  task  situation  as  well  as  an  overview 
of  essential  theory.  Supplementary  media  for  these  lessons  are  in  the  form 
of  overhead  projector  transparencies. 

Practice  lessons  involve  "hands-on"  performance  training  in  the  Cockpit 
Procedures  Trainer  (CPT)  for  the  pilots  and  in  the  Weapon  Systems  Trainer 
(WST)  for  the  NFOs/FTs.  These  lessons  involve  psychomotor  skills  development 
which  builds  upon  the  prerequisite  cognitive  lessons.  A  fully  proceduralized 
CPT  Syllabus  manual  (based  upon  the  Behavioral  Objectives)  was  developed  to 
support  these  exercises  for  pilot  training.  Similarly,  step-by-step  directions 
were  developed  in  the  form  of  individual  student  handouts  for  NFO/FT  practice 
training. 

Sortie/Scenario  lessons  draw  upon  the  knowledges  and  skills  toward  which 
the  cognitive  and  practice  lessons  are  directed.  Simulation  of  actual  mission 
performance  is  provided  in  units  of  increasing  complexity.  These  lessons  are 
conducted  in  the  Operational  Flight  Trainer  (OFT)  and  the  E-2  aircraft  for  the 
pilots  and  in  the  WST  and  the  E-2  aircraft  for  the  NFOs/FTs.  Training  materials 
developed  for  these  exercises  consist  of  Flight  Manuals  for  both  pilots  and  NFOs. 

Block  flow  charts  in  Figures  6  through  9  describe  the  primary  ISD- 
based  pilot  and  NFO/FT  syllabi  and  associated  collateral  courses.  These 
flow  charts  present  linear  tracks,  with  secondary  levels  indicating  lesson 
sequences  with  interchangeable  ordering.  The  sequence  is  ordered  by  lessons 
represented  by  connecting  blocks  and  organized  into  modules.  Each  lesson  block 
in  the  flow  charts  identifies  the  content,  the  medium,  and  the  nominal  in¬ 
structional  time  to  criterion.  As  a  general  rule,  the  recommended  sequence 
by  lesson  and  by  module  must  be  followed,  but  with  two  exceptions: 

(1)  If  a  trainee  reaches  criterion  on  the  entire  cognitive  sequence  of  a 

module  and  then  finds  the  trainer  unavailable  for  a  practice  or  sortie/ 
scenario  lesson,  he  may  be  allowed  to  proceed  into  the  cognitive  lessons 
in  the  next  module; 
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(2)  Aircraft  lessons  may  be  completed  up  to  two  modules  behind  schedule, 
but  may  not  be  performed  prior  to  completing  all  cognitive,  practice 
and  sortie/scenario  lesson  prerequisites.  Additionally,  it  has  been 
determined  that  the  Field  Carrier  Landing  Practice  (FCLP)  and  Carrier 
Qualification  (CQ)  modules  in  the  pilot  syllabus  may  be  undertaken 
earlier  in  the  sequence  (after  Module  XII)  to  meet  carrier  deck 
availability  if  required. 

The  above  described  syllabi  are  for  trainees  in  Categories  I  and  II. 

However,  the  modular  structure  allows  for  a  reduced  syllabus  to  meet  the 
needs  of  trainees  in  special  categories.  This  provision  has  been  used  to 
successfully  accommodate  E-2B  to  E-2C  transition  classes.  It  should  be  noted 
also  that  the  FT  syllabus  eliminates  the  five  tactical  modules  identified 
as  the  "OPS"  phase. 

Tables  1  and  2  further  describe  the  syllabi  by  hours  in  each  medium 
of  the  ISD-based  syllabus.  Additionally,  the  time  for  collateral  courses  is 
indicated.  The  entire  curriculum  can  be  completed  in  approximately  16  weeks 
for  pilot  training,  and  26  weeks  for  NFO  training. 

TESTINC.  Each  lesson  in  the  training  syllabus  is  concluded  with  a  performance- 
oriented,  criterion-referenced  test.  All  test  items  are  derived  from  the  Behav¬ 
ioral  Objectives.  Cognitive  testing  utilizes  chemically-treated,  self-scoring 
answer  sheets  in  the  multiple-choice  format.  Practice  and  Sortie/Scenario 
testing  relies  on  subjective  assessments  which  are  standardized  through  the  use 
of  a  set  of  criteria  derived  from  the  Behavioral  Objectives.  The  entire  system 
is  designed  to  assure  and  certify  criterion  proficiency  on  every  Behavioral  Ob¬ 
jective  prior  to  course  completion.  To  record  progress  toward  this  proficiency, 
Student  Record  Sheets  have  been  designed  on  a  modular  basis.  Figure  10 
presents  an  example  from  the  pilot  syllabus.  Entries  are  made  on  time  and  number 
of  errors  to  criterion  for  cognitive  testing  and  pass/fail  or  "0-4"  scale  scores 
for  Practice  and  Sortie/Scenario  testing. 

Viewing  the  cognitive  testing  procedure  from  the  student's  perspective, 
the  process  is  as  follows: 

a.  The  student  proceeds  at  his  own  pace  in  learning  the  information 
presented  in  a  sound-slide  lesson. 

b.  When  the  student  is  satisfied  that  he  has  mastered  the  objectives 
covered,  he  checks  out  a  form  of  the  corresponding  cognitive  test. 

c.  The  student  then  attempts  to  answer  a  series  of  multiple  choice 
questions  by  selecting  the  correct  response  on  a  self-scoring  answer  sheet. 

d.  As  the  student  marks  the  chemically-treated  coating  of  the  answer 
sheet,  he  is  given  immediate  feedback. 

e.  If  he  chooses  an  incorrect  alternative,  he  continues  to  respond  until 
correct. 

f.  The  student  reviews  his  errors  and  the  corresponding  instructional 
material. 

g.  Finally,  he  consults  with  the  Learning  Center  Supervisor  who  prescribes 
remediation  and/or  tutors  him  until  reaching  the  criterion  performance  level. 
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TABLE  2.  ISD-BASED  NFO/FT  SYLLABUS  BY  HOURS  IN  MEDIA 

(May  1978) 


m 


I 
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TABLE  2.  (CONTINUED) 


MOlUiLl. 

CARR!  1. 

CLASSROOM 

1  51-8 

L-2 

n.xr 

TOTAL 

XXVI 

4 

(> 

10.0 

XXV  I  1 

a 

6 

10.0 

XU  1  1  1 

4 

6 

10.0 

XXIX 

4 

6 

10.0 

XXX 

1 

3 

7.0 

"3.0/74.0 

32.  n 

66. 0 

^9.  0 

1  5 . 0 

235. '7  234.0 

1 - - 1 

*  INATOPS)  1  Hour  in  CRL  and  2  hours  TI.ST 
**  YU  Cockpit  i AM  =  1.3  hours  CRT 


COLLATF.RAL  COUR.3L.3  (Secondary  Tracks) 


Aircraft  Systems 

5 . 5 

NFO/PT 

General  Military  Training 

37 . 5 

NF'O/FT 

Required  Reading 

NFO/ FT 

li.  W.  Classes 

io.  o 

NF0/FT 

SURE  School 

40.0 

N'FO/FT 

Fire  Fighting 

20.0 

NF0/FT 

Aviation  Physiology 

20.0 

NFO/FT 

L'lC  Air/Surf.  Coord. 

40.0 

MFO 

Inst  Ground  School 

20.0 

NFO 

AIC 

240.0 

NFO 
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Practice  and  Sortie/Scenario  testing  is  conducted  by  the  Trainer  and 
Aircraft  Instructors  with  items  prescribed  via  a  checklist  format.  An  example 
of  the  grading  criteria  from  the  pilot  syllabus  is  presented  below. 

PILOT  GRADING  CRITERIA 


0  —  Demonstrates  lack  of  knowledge  and/or  prerequisite 

skills  necessary  to  perform  objectives. 

1  —  Demonstrates  knowledge  of  procedures,  but  performs  objectives 

below  the  stated  criterion  with  instructor  assistance. 

2  —  Performs  objectives  to  the  stated  criterion  level  with  in¬ 

structor  assistance. 

3  —  Performs  objectives  to  the  stated  criterion  level  without 

instructor  assistance. 

4  —  Performs  objectives  with  accuracy  and  efficiency  superior  to 

the  criterion  level  without  instructor  assistance. 

These  criteria  are  applied  in  variable  fashion  with  #3  as  the  minimum  graduating 
level.  Grades  1,  2,  3  and  4  are  referenced  to  the  performance  limits  stated  in 
the  appropriate  Behavioral  Objectives.  The  grading  criteria  are  supplemented 
with  discrepancy  checks  for  Basic  Air  Work  and  Procedures. 

MEDIA.  The  general-purpose  audio-visual  carrel  is  utilized  extensively  through¬ 
out  the  pilot  and  NFO/FT  syllabi  as  the  medium  through  which  the  individualized 
cognitive  instruction  is  presented.  Cognitive  lessons  for  both  pilot  and  h'FO/FT 
trainees  are  predominantly  oriented  toward  the  teaching  of  the  knowledge  necessary 
to  operate  the  aircraft  systems.  The  function  of  the  general-purpose  carrel  in 
Figure  II  is  to  provide  student-centered,  individualized  instruction  that  does 
not  require  active  hands-on  training.  The  carrels  incorporate  an  audio-visual 
presentation  and  a  work  space  for  writing.  The  audio-visual  presentations  consist 
of  a  student-paced,  narrated,  slide  presentation  that  is  linearly  programmed  (i.e., 
without  branching  logic)  but  with  a  student-initiated  synchronized  review  capability. 
At  RVAW-120  there  are  a  total  of  19  carrels,  of  which  7  are  installed  in  the  Pilot 
Learning  Center  and  11  in  the  NFO  Learning  Center.  The  remaining  carrel  has  been 
installed  in  the  Training  Development  Department  (TDD)  office  for  use  by  TDD 
personnel  in  the  ongoing  evaluation  and  revision  program. 

Hands-on  training  devices  are  utilized  for  both  pilot  and  NFO/FT  training 
programs.  These  devices  have  been  integrated  throughout  both  the  pilot  and  NFO/FT 
syllabi  in  such  a  manner  as  to  provide  early  hands-on  training.  There  are  three 
ground-based  training  devices,  one  NFO  and  two  pilot,  through  which  hands-on 
training  may  be  conducted.  NFO  hands-on  training  is  accomplished  through  the  use 
of  the  Tactics  Trainer,  device  15F8  (Figure  12).  Pilot  hands-on  training  utilizes 
both  the  Cockpit  Procedures  Trainer,  device  2C20  (Figure  13)  and  the  Operational 
Flight  Trainer,  device  2F110  (Figure  14).  The  E-2C  aircraft  itself  (Figure  15) 
is  also  used  for  both  Pilot  and  NFO/FT  training. 
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Figure  12.  NFO/FT  Tactics  Trainer 
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The  15F8  Tactics  Trainer  is  a  non-motion-base  sophisticated  state-of-the- 
art  training  device  utilized  by  both  NFO's  and  FT's  to  fulfill  practice  and 
scenario  training  requirements.  The  Tactics  Trainer,  which  fully  simulates  the 
E-2C's  CIC  compartment,  is  capable  of  presenting  preprogrammed  scenarios  to  the 
trainees  as  well  as  full  video,  symbol  and  system  alpha  numeric  generation  for 
all  modes  of  system  operation.  The  simulated  CIC  compartment  contains  three 
student  consoles:  one  for  each  of  the  CICO,  ACO  and  FT  stations.  Also  included 
within  the  simulated  CIC  compartment  is  a  functional  computer  cabinet  (OL-77) 
for  the  loading  of  standard  tactical  programs.  Training  may  be  accomplished  on 
an  individual  or  team  training  basis  on  any  one  of  the  three  student  consoles. 
Support  instructor  personnel  will  vary  in  accordance  to  the  complexity  of  the 
tactical  problem  being  run.  Module  practice  sessions  normally  require  between 
two  and  three  support  instructor  personnel  for  student  instruction  and  instructor 
console  operation.  The  operationally  oriented  module  scenarios  require  approxi¬ 
mately  four  instructors,  and  Type  Training  (TYT)  exercises  and  Operational  Read¬ 
iness  Evaluation  (ORE)  scenarios  require  the  full  complement  of  seven  instructors. 
The  NFO/FT  syllabus  includes  30.0  Tactics  Trainer  hours  devoted  to  module  practice, 
9.0  hours  of  module  scenarios,  24.0  hours  of  TYT  exercises,  and  3.0  hours  for  ORE 
scenario  training. 

The  Cockpit  Procedures  Trainer  (CPT)  is  a  computer-supported,  fixed-base 
cockpit  replica  trainer  which  is  uilized  in  the  instruction  and  practice  of 
both  normal  and  emergency  cockpit  procedures.  A  maximum  of  two  students  may  be 
accommodated  during  the  practice  sessions.  One  instructor  is  required  to  provide 
supervision  and  instructor  console  operation.  With  the  advent  of  the  ISD  syllabus, 
a  modification  to  the  CPT  has  been  made  which  allows  for  the  use  of  audio/visual 
aids  in  conjunction  with  CPT  operation.  This  modification  provides  the  student 
at  the  pilot's  station  with  the  same  audio/ visual  capability  achieved  with  the 
general-purpose  carrels.  Additionally,  the  CPT  audio/visual  lessons  were  designed 
so  that  they  may  be  utilized  as  "stand-alone"  lessons  in  the  learning  center,  should 
the  CPT  be  unavailable  or  require  maintenance.  Trainees  are  scheduled  for  CPT 
practice  sessions  upon  completion  of  the  cognitive  lessons  within  a  module.  The 
pilot  syllabus  includes  11.0  hours  of  CPT  instruction;  19.0  additional  hours 
augmented  with  audio/ visual  support.  CPT  instruction  begins  with  the  initial  flight 
phase  and  covers  all  phases  of  flight  procedures  training  through  the  NATOPS  qual¬ 
ification  phase. 

The  E-2C  Operational  Flight  Trainer  (OFT)  is  a  major  state-of-the-art  trainer 
incorporating  features  to  ensure  realism  and  fidelity,  training  versatility,  and 
pilot  performance  monitoring  and  recording.  Realism  is  enhanced  through  faithful 
cockpit  reproduction:  A  six-axis,  six-degree-of-f reedom  motion  base  to  provide 
proprioceptive  cues.  A  visual,  computer  generated  image  system  will  provide  carrier 
as  well  as  field  landing  simulation.  Aural  simulation  will  also  be  extremely 
realistic  through  the  reproduction  of  normal  aircraft  sounds  which  include  engines, 
hydraulics,  air  stream  and  even  ground  power  carts.  Additionally,  ground  or 
carrier  deck  handling  simulation,  environmental  conditions  such  as  wind,  rain  and 
icing,  as  well  as  prerecorded  ATIS  broadcasts,  complete  the  simulation  and  immerse 
the  pilots  in  the  training  sortie.  Also  noteworthy  is  the  fact  that  there  will  be 
more  than  150  available  .malfunctions  which  will  allow  flight  crews  to  see  and  deal 
with  many  situations  which  have  heretofore  been  only  discussed,  because  simulation 
in  the  aircraft  was  not  possible  or  too  dangerous.  OFT  flights  account  for  37 
hours  within  the  pilot  syllabus  and  will  cover  all  aspects  of  aircraft  flight  and 
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handling  characteristics  from  the  initial  flight  phases  through  the  carrier 
qualifications  phase. 

Finally,  the  ultimate  in  hands-on  training  devices  is,  of  course,  the 
E-2C  aircraft.  The  aircraft  is  considered  to  be  a  part  of  the  media  support 
because  of  the  integrated  flight  syllabus .for  both  Pilots  and  NFO/FT's. 

Through  the  use  of  the  integrated  flight  syllabus,  the  traditional  sequence 
of  "ground  school/flight  syllabus"  has  been  altered  to  provide  early,  optimally- 
spaced  hands-on  training  throughout  the  entire  syllabus.  The  pilot  syllabus 
contains  a  total  of  69.0  flight  hours  which  are  utilized  to  complete  the  entire 
flight  syllabus.  Of  these  hours,  36.0  are  devoted  to  twelve  aircraft  flights, 
which  begin  with  basic  aircraft  familiarization  and  conclude  with  NATOPS 
standardization.  The  remaining  33.0  hours  are  divided  into  field  carrier  land¬ 
ing  practice  flights  (20.0  hrs)  and  carrier  qualification  flights  (13.0  hrs.). 

The  NF0  flight  syllabus  is  also  integrated  and  contains  a  total  of  20 
flight  hours.  It  should  be  noted  that  there  are  15  flight  hours  not  reflected 
within  the  above  figure  that  are  dedicated  to  AIC  training,  which  is  considered 
to  be  a  collateral  course  for  the  NFO's.  The  NFO  flight  syllabus  consists  of 
9  flights,  beginning  with  Radar  and  terminating  with  NATOPS  standardization. 
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COMPUTER  DATA  BASE 

The  Instructional  Systems  Development  Impact  System  Directory  (ISDISD)  was 
designed  to  facilitate  a  process  of  continuous  revisions  to  the  E-EC  curriculum. 

By  incorporating  all  ISD  analytical  documentation  as  well  as  source  and  curriculum 
documentation,  the  system  provides  a  means  of  ensuring  accuracy,  completeness ,  and 
conformity  to  the  systems  approach  to  training.  An  overview  of  its  design  capability 
and  examples  of  the  data  encoding  formats  used  for  the  E-EC  documentation  are 
given  below. 

The  ISDISD  is  a  computer-managed  information  system  which  provides  a  directory 
of  tasks,  behavioral  objectives,  lessons,  and  hardware/software  data.  Numerous 
computer  programs  support  and  provide  a  means  of  maintaining,  modifying,  and 
displaying  information  pertaining  to  the  hierarchically-structured  Task  Analysis 
Data  Base,  Behavioral  Objectives  Data  Base,  Lesson  Data  Base,  and  supplemental 
Hardware/Software  Data  Bases  (NATOPS  Flight  Manual,  Supplemental  NATOPS  Flight 
Manual,  Software  Manual,  Tactics  Manual,  and  Work  Unit  Code  Manual).  The  structure 
of  the  three  primary,  on-line  data  bases  (i.e.,  the  Task  .Analysis  Data  Base,  the 
Behavioral  Objectives  Data  Base,  and  the  Lesson  Data  Base)  is  a  reflection  of  the 
Systems  Approach  to  Training  (SAT).  As  such,  materials  found  in  each  succeeding 
data  base  are  derived  from  materials  provided  by  the  preceding  data  base.  In 
addition  to  this,  the  Hardware/ Software  Data  Bases  provide  the  Task  Analysis  Data 
Base  with  the  necessary  information  concerning  the  E-E  system.  Referring  to 
Figure  16,  the  hierarchical  arrangement  of  the  on-line  data  bases  and  the 
manner  in  which  the  Hardware/ Software  Data  Bases  impinge  upon  the  Task  Analysis 
Data  Base  is  schematically  displayed.  Additionally,  this  figure  shows  the  linkages 
existing  between  the  data  bases  and  the  subcategories  of  each  data  base. 

The  ISDISD  has  the  capability  of  displaying  and/or  printing  both  the  in¬ 
formation  contained  in  these  data  bases  and  the  linkages  between  these  data  bases. 
This  is  accomplished  through  an  interactive,  on-line  query  process.  Given  a 
change  of  information  in  any  of  the  system's  data  bases,  whether  proposed  or 
actual,  ISDISD  will  answer  the  following:  What  tasks  are  affected?  WThat  questions 
and  objectives  for  those  lessons  are  affected?  What  media,  lesaon  plans,  and  other 
course  materials  are  affected?  Where  is  the  media  located  in  the  master  file 
library  so  that  it  can  be  reviewed? 

The  information  needed  to  build  the  data  base  can  be  placed  on  computer 
magnetic  tape,  9-track  800-1600  BPI,  ASCII  format,  and  batch  processed.  Whereas 
this  can  be  done  through  an  on-line  interactive  system,  the  capability  does  exist 
to  place  the  information  on  a  tape  cassette  generated  by  a  Xerox  800  word  process¬ 
ing  system.  The  elements  in  each  data  base  are  placed  sequentially  on  magnetic 
tape,  reading  the  data  bases  hierarchical  structure  top  to  bottom,  left  to  right. 

On  the  magnetic  tape,  this  hierarchical  structure  is  seen  in  the  left  to  right 
coding  of  data  fields.  The  starting  point  and  ending  point  of  a  data  field  is 
indicated  by  the  placement  of  an  asterisk  at  the  start  and  finish  of  each  field. 

Each  level  of  the  data  base's  hierarchical  structure  is  comprised  of  two  fields. 

The  first  field,  initiated  by  the  appearance  of  a  letter  and  number,  signifies 
the  level  being  described  and  the  number  associated  with  the  specific  element 
identified  at  that  level.  This  field  is  followed  by  an  asterisk.  Following  this 
asterisk,  the  next  field  contains  a  brief  description  or  name  referring  to  the 
letter  and  number  found  in  the  first  field.  An  asterisk  is  placed  at  the  end 
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Figure  16.  ISDISD  Data  Bases 
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of  this  descriptor,  signifying  the  end  of  the  second  field  ind  the  beginning 
of  the  third.  These  two  fields,  (1)  the  letter  end  n nr. her,  one  i 2)  the 
descriptor,  comprise  the  information  needed  to  describe  earn  level  f  ,r  a  given 
data  base.  The  rest  of  the  data  base's  levels  are  described  in  a  similar 
fashion • 

Using  the  Task  Analysis  Data  Base  as  in  example,  the  c  .rr<'s;e,nd ;  ng 
symbology  and  field  devices  are  described  below: 


FIELD  TITLE 

HIERARCHICAL  STR 

JOB 

J •/*  DESCRIPTION* 

RESPONSIBILITY 

Rl** 

MISSION 

M'»*  DESCRIPTION* 

PHASE 

PI** 

TASK 

T'/*  DESCRIPT  ION* 

SL'BTASK 

STi** 

BEHAVIOR  STATEMENT 

BS*  DESCRIPTION* 

CONDITIONS  STATEMENT 

CS*  DESCRIPTION* 

STANDARDS  STATEMENT 

SS*  DESCRIPTION* 

TRAINING  REQUIRED 

T  Ot* 

WHERE  TAUGHT 

W  Tit* 

CLASSIFICATION 

cot* 

CRITICALITY  LEVEL 

cut* 

To  an  initial  on-line  query,  the  computer  will  print  out  information  up 
to  and  including  the  TASK  level. 

An  example  of  placing  the  records  on  a  tape  would  be  as  follows: 

Jl*  description*  (NOTE:  the  asterisk  starts  and  ends  a  field)  Rl* 
description*  Ml*  description*  PI*  description*  Tl*  description*  BS*  description* 
CS*  description*  SS*  description*  TC3*  WT1*  CC2*  CL3*  (the  next  record  would 
start  here)  9T2*  description*  BS,  CS,  etc.,  for  this  task  T3*  description* 
(additional  data)  P2*  description*  \it  description*  (additional  data)  etc.  In 
other  words,  given  a  J it,  Rit,  M it,  and  Pit,  every  task  would  be  listed  under  that 
P#;  then  the  next  Pit  and  all  the  task  under  that,  until  the  phases  were 
completed;  then  a  new  M#,  the  first  Pit  under  that;  all  the  Tit  under  that  Pit; 
next  Pit,  etc.,  etc. 
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Identical  tasks  and  sub-tasks,  with  their  related  data,  are  often  used  in 
various  places  in  the  task  track.  ISDISD  provides  a  method  to  handle  this 
situation  without  duplication  of  data.  For  example,  a  task  and  its  related 
data  that  has  already  been  placed  in  the  data  base  can  be  referenced  by  this 
format  when  entering  a  new  task:  J it*  R it*  Hit*  Pit*  T It*  X  J it*  R It*  Hit*  P it*  Ttt*. 

If  a  new  task  is  added,  it  would  be  normal  to  have  the  description  after  the 
*  of  the  T #.  However,  in  this  case,  X  after  the  *  tells  the  computer  program 
that  it  is  a  cross-referenct  to  a  prior  J // ,  R#,  M#,  Pit,  and  T it.  Pointers  will 
then  be  set  up  by  the  computer  to  point  to  the  Tit  description,  BS,  CS  and  SS 
description,  and  TC#,  WT# ,  CCIt  and  CL#  of  the  original  record.  Any  number  of 
cross-references  can  be  made  to  the  original  record.  If  the  original  record 
is  deleted,  it  is  cross-referenced  by  other  T#'s.  The  basic  device  for  cross- 
referencing  is  the  pointer  or  index  number.  These  consist  of  a  series  of  the 
first  field,  i.e. ,  letter  and  number,  caricatures  for  each  level  of  the  data 
base.  Whereas  those  pointers  could  conceivably  index  all  levels  of  the  data, 
this  is  not  the  case.  Pointers  index  only  to  the  level  at  which  linkages  between 
linking  the  Task  Analysis  and  Lesson  Data  Bases  occur  at  the  TASK  level  of 
the  former  and  the  LESSON  level  of  the  latter. 

In  that  the  ISDISD  is  a  computer-based  system,  it  easily  manages  the  complex 
Instructional  Systems  Development  (ISD)  training  program.  It  effectively  deals 
with  a  multitude  of  varied  changes  that  could  affect  any  data  base  and  impact 
the  instructional  process.  With  the  capability  of  providing  either  on-line  or 
print-out  information  for  all  the  data  bases,  this  system  facilitates  the 
maintenance  and  modification  of  ISD  data  bases.  A  major  benefit  of  this  system 
is  that  it  does  not  place  any  restrictions  on  any  of  the  data  bases  as  to  the 
depth  of  the  hierarchically  structured  levels,  and  it  allows  the  use  of  any 
numbering  system  to  identify  elements  at  the  different  levels  of  each  data  base. 
Additionally,  the  combination  of  the  ISD's  hierarchical  structure  and  its  coding 
system  offers  a  viable  device  by  which  the  various  data  bases  can  be  cross- 
referenced. 

While  the  ISDISD  data  bases  were  encoded  for  the  specific  use  of  facilitating 
the  maintenance  and  revision  of  the  E-2C  ISD  program,  the  conceptualization  and 
the  supporting  computer  programs  were  designed  for  a  more  generic  application. 

All  ISD  programs  developed  in  accordance  with  the  MIL-T-29053  specification  can 
make  use  of  these  programs  and  data  encoding  formats.  In  fact,  the  goal  of  generic 
utility  was  the  primary  factor  in  the  design  of  the  ISDISD,  thus  requiring  the 
adaptation  of  the  E-2C  specific  data  to  fit  the  specification's  framework. 

Necessary  adaptations  were  accommodated  through  the  use  of  "dummy  statements"  and 
the  restructuring  of  content  within  the  contractor's  Phase  I  data  bases. 
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Evaluation  In  a  comprehensive  ISD-based  program  must  be  directed  toward 
both  formative  and  summative  measures.  To  be  operationally  feasible,  these 
measures  must  then  be  integrated  into  an  organized  data  base.  Finally, 
evaluation  must  be  tailored  to  the  particular  course  structure  and  learning 
strategies  inherent  in  the  system  under  analysis.  Such  an  evaluation  system 
was  designed  for  the  E-2  aircrew  training  program.  Measures  of  learning 
acquisition,  retention,  and  transfer  were  integrated  into  a  progression  of 
cognitive,  practice,  and  sortie/scenario  type  lessons.  The  evaluation  system 
then  provided  for  a  sophisticated  computer  data  base,  the  Aviation  Training 
Support  System  with  associated  analytical  programs.  Figure  17  presents  a  model 
of  the  information  flow  within  this  evaluation  system.  Formative  Evaluation 
is  the  initial  development  component  which  consists  of  Quality  Control  and 
Revision  input  to  the  Curriculum.  Quality  Control  procedures  involve  the 
application  of  acceptance  checklists  to  storyboard  or  other  preliminary  draft 
instructional  materials,  while  revision  is  based  upon  lesson,  modular,  and 
course  level  data  from  materials  tryout.  Summative  Evaluation  provides 
Quality  Control  input  to  the  Curriculum  via  Revision  after  the  program  has 
been  implemented  in  its  entirety.  This  component  of  the  overall  evaluation 
model  serves  the  functions  of  both  an  internal  and  an  external  validation 
of  the  curriculum  as  well.  The  Aviation  Training  Support  System  provides  the 
computer  data  base  component  of  the  evaluation  model  which  forms  the  heart 
of  the  system.  This  data  base  begins  with  all  ISD  documentation  leading  to 
initial  development  of  the  curriculum  and  is  expanded  to  include  updates 
from  system  changes.  An  additional  function  of  the  ATSS,  although  not  opera¬ 
tional,  would  provide  revision  from  Formative  Evaluation  and  Summative  Eval¬ 
uation.  These  data  sources  are  processed  by  the  ATSS  which  documents  and 
facilitates  curriculum  revisions  and  has  the  capability  of  providing  valida¬ 
tion  reports  to  training  management.  The  evaluation  model  presented  in 
Figure  17  is  further  defined  by  a  performance  assessment  strategy  illustrated 
in  Figure  18.  This  strategy  applies  to  both  formative  and  summative  eval¬ 
uation,  with  differences  occurring  only  in  data  collection  techniques  and 
areas  of  emphasis.  Each  component  of  the  strategy  is  briefly  described  below. 

Formative  Evaluation  is  applied  to  the  three-part  training  strategy  in¬ 
volving  cognitive,  practice,  and  sortie/scenario  lessons  in  a  continuum  from 
learning  acquisition  to  retention  and  transfer.  In  consonance  with  the  per¬ 
formance-oriented  approach  to  aircrew  training  upon  which  the  E-2  ISD  program 
has  been  based,  learning  is  evaluated: 

(1)  in  the  cognitive  setting  where  the  academic  prerequisites  of  a 
skill  are  developed; 

(2)  in  the  practice  setting  where  the  actual  component  skills  are 
developed ; 

(3)  in  the  sortie/scenario  setting  where  skills  are  integrated  and 
applied;  and 

(4)  in  the  broader  setting  of  course  level  objectives  and  criteria. 

I 
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•Sumrmtive  Evaluation  only 


Figure  18.  Performance  Assessment  Strategy:  Formative  and  Summative  Evaluation 
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The  purpose  of  the  overall  Formative  Evaluation  is  therefore  a  method  of 
collecting  and  organizing  data  which  contributes  to  curriculum  development  at  the 
lesson  level,  but  also  at  the  modular,  supra-modular ,  and  course  level.  Thus,  no 
data  is  viewed  only  in  isolation.  Rather,  it  is  available  for  a  systems  approach 
to  analysis. 

Summative  Evaluation  is  best  viewed  as  a  dual-purpose  component  of  the 
overall  evaluation  system.  One  purpose  is  to  provide  training  program 
validation  against  base-line  data  from  the  former  training  program  and  the 
fleet.  The  second  purpose  is  to  provide  input  to  Management  Feedback  and 
Quality  Control.  The  former  purpose  is  achieved  as  the  end-product  of  the 
fully  implemented  program  and  requires  a  comprehensive  one-time  data 
collection  effort.  The  latter  purpose  requires  a  continuing  activity  and 
is  one  that  periodically  results  in  reports  to  management  and  revision  to 
the  curriculum.  The  Summative  Evaluation  strategy  can  be  compared  to  the 
previously  presented  Formative  Evaluation  strategy,  as  the  basic  structure 
of  learning  types  and  learning  sequences  remains  essentially  the  same. 

The  only  difference  in  evaluation  structure  lies  in  the  absence  of  the 
Retention  Component.  While  this  component  remains  of  interest  to  Summative 
Evaluation,  it  is  subsumed  under  the  Practice  and  Sortie/Scenario  components 
where  it  takes  on  an  adaptive  function.  The  Summative  Evaluation  represents 
a  shift  not  only  from  small  group  to  large  group  data  collection,  but  also 
from  materials  tryout  to  student  performance.  The  former  emphasis  provided 
by  Formative  Evaluation  contributes  to  curriculum  development  (including 
initial  revision),  while  the  latter  emphasis  contributes  to  validation, 
revision,  and  maintenance.  Both  Formative  and  Summative  Evaluation  are 
necessary,  however,  and  can  be  particularly  useful  when  integrated  by  such 
a  Training  Feedback  mechanism  as  the  Aviation  Training  Support  System. 

Data  collection  instruments  and  procedures  were  developed  by  the 
contractor  and  the  SMEs  to  fulfill  the  requirements  of  both  formative  and 
summative  evaluation.  These  instruments  can  be  characterized  as  checklists, 
questionnaires,  and  tests.  Questionnaires  involved  squadron  and  fleet  feed¬ 
back  instruments  as  well  as  student  attitude  surveys.  Test  development 
procedures  were  designed  to  assure  a  criterion-referenced,  performance-based 
orientation.  The  output  of  the  overall  data  collection  design  was  then  in¬ 
tegrated  into  a  scheme  for  providing  Revision  Specifications. 
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An  essential  requirement  of  any  comprehensive  ISD  effort  is  the  preparation 
of  a  logical  and  orderly  implementation  plan.  The  plan  must  focus  on  training 
management  and  administration  personnel,  procedures,  and  techniques.  Such  a  plan 
provided  the  ISD  management  personnel  with  an  overview  of  the  proposed  in¬ 
structional  systems,  instructional  systems  integration  procedure  and  schedules, 
specifications  for  the  resources  and  operation  of  the  Student  Learning  Centers 
(SLC),  and  role  descriptions  of  training  personnel.  Feasibility  as  well  as 
optimum  instructional  effectiveness  were  primary  considerations  in  the  formula¬ 
tion  of  these  plans.  Operational  considerations  were  carefully  incorporated  to 
ensure  a  smooth  transition  from  the  existing  to  the  revised  instructional  systems. 
These  considerations  were  based  on  the  best  available  projection  data  and  were 
designed  to  accomodate  the  necessary  flexibility  within  the  range  of  normal 
operations.  Then,  too,  potential  constraints  were  identified  and  contingency  plans 
were  incorporated. 

The  implementation  schedule  began  with  projected  completion  dates  for  the 
developmental  tasks  and  then  provided  for  instructor  training  dates,  class  conven¬ 
ing  dates,  and  evaluation/ revision  task  periods.  Milestones  were  then  established 
for  facility  and  equipment  readiness  and  "kickoff"  meetings. 

Specifications  for  the  SLC  were  among  the  most  important  components  of  the 
plan.  Figure  19  presents  a  conceptualization  of  the  proposed  squadron  organiza¬ 
tion  as  it  impacts  on  personnel  and  facilities  operations.  The  hub  of  the  SLC  is 
the  Student  Records  Section  through  which  all  trainee  activities  are  coordinated. 
Primary  functions  of  the  Student  Records  Section  are: 

(1)  to  issue  daily  and  weekly  schedules  (prepared  by  the  Scheduling 
Section) ; 

(2)  to  issue  instructional  materials  (including  tape/slide  lessons); 

(3)  to  maintain  individual  and  master  records  of  student  performance 
in  the  carrel,  classroom,  trainers,  and  aircraft  for  tracking 
and  evaluation  purposes;  and 

(4)  to  ensure  that  each  student  progresses  through  the  prescribed 
course  on  schedule  and  in  the  proper  sequence. 

The  Scheduling  Section  has  the  primary  function  of  preparing  daily  and  weekly 
schedules  for  aircraft  flights,  trainers,  and  classroom  sessions.  Detailed  schedul¬ 
ing  of  lessons  designated  for  audio-visual  carrels  is  not  required.  Instead,  each 
student  is  responsible  (as  directed  by  the  Student  Records  Section)  for  individual¬ 
ly  completing  the  carrel  lessons  which  are  prerequisite  to  the  aircraft,  trainer, 
and  classroom  lessons.  Thus,  close  coordination  between  the  Student  Records  Section 
and  the  Scheduling  Section  is  essential.  A  series  of  tables  indicating  expected 
student  progress  by  media  and  training  day  provide  assistance  in  scheduling. 
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STUDENT  LEARNING  CENTER 


Figure  19.  Instructional  System  Components 
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Traditional  instructor  roles  are  described  for  the  Classroom,  Trainer,  and 
Aircraft  components  of  the  system.  However,  the  A-V  Center  component  is  an  in¬ 
novation  to  E-2  training  procedures.  The  A-V  Center  provides  the  environment  for 
the  primary  instructional  medium,  the  tape/slide.  Within  this  environment,  the 
student  is  seated  at  an  individual  carrel,  where  he  proceeds  through  tape/slide 
lessons  at  his  own  pace.  Upon  completion  of  a  tape/slide  lesson,  the  student 
must  reach  criterion  on  a  multiple-choice  test  before  proceeding  to  another 
lesson.  Fully  qualified  instructors  designated  as  Learning  Center  Supervisor 
(LCSs)  must  be  on  duty  within  the  A-V  Centers  throughout  the  training  day.  The 
role  of  the  LCS  is  to  assist  the  students  with  any  learning  problems  encountered 
while  viewing  a  tape/slide  lesson  (e.g.  answer  questions,  provide  clarification 
and  amplification,  etc.)  as  requested.  More  importantly,  however,  he  reviews 
all  incorrect  test  responses  with  the  student  and  provides  remediation  as 
necessary.  The  LCS  further  provides  reports  to  the  ISD/Training  Development 
Departments  on  problems  encountered  with  the  training  system. 

INSTRUCTOR  TRAINING 

A  prime  responsibility  of  the  contractor  was  the  training  of  SMEs 
for  their  roles  in  the  authoring  and  revision  tasks  and  the  instructors  for 
their  roles  in  the  operational  instructional  system.  The  former  requirement 
was  satisfied  with  the  contractor's  reference  materials  (i.e.  "Guidelines 
and  Readings  in  Instructional  Technology"  and  "Instructional  Technology  for 
Aircrew  Training")  and  on-the-job  training  under  the  supervision  of  the 
contractor's  IPs,  media  specialists,  and  technical  writers.  These  materials 
outlined  the  procedures  for  the  entire  ISD  process,  but  focused  heavily  on 
"storyboarding"  techniques  for  audio-visual  presentations  and  test  develop¬ 
ment  procedures  for  multiple-choice  items.  The  latter  requirement,  instructor 
training,  was  fulfilled  via  a  seminar  conducted  by  the  contractor's  IPs  and 
training  analysts.  The  seminar  was  centered  around  a  lecture  guide  supported 
by  overhead  projector  transparencies  and  individual  student  handouts.  While 
the  handouts  incorporated  the  previously  described  SME  authoring  guidelines 
(primarily  as  self-study  and  reference  materials),  they  and  the  seminar  were 
more  heavily  oriented  toward  the  roles  of  the  instructors  and  the  operation 
of  the  training  system. 

Table  3  presents  the  Table  of  Contents  from  the  Seminar  Instructors 
Guide.  The  first  four  sections,  "Analysis  and  Design  of  ISD-Based  Training 
Program,"  "Training  System  Description,"  "Training  Operations,"  and  "Performance 
Evaluation  and  Revision"  are  keyed  to  the  needs  of  the  instructor.  Detailed 
instruction  in  these  areas  and  an  overview  of  the  development/revision  proce¬ 
dures  precedes  further  self-study  and  on-the-job  training.  A  fully  qualified 
instructor  must  also  become  familiar  with  the  content  of  the  individualized 
aircrew  training  lessons,  and  must  complete  a  standardization  syllabus  for 
trainer  and  flight  exercises. 

During  the  follow-on  revision  period,  the  instructor  training  syllabus  was 
revised  and  expanded  to  include  the  following  slide/tape  presentations: 

"ISD  Overview,"  "Instructor  Roles,"  Training  Operations,"  "Quality  Control  and 
Revision,"  "Conducting  Trainer  and  Flight  Exercises,"  "Lesson  Authoring," 
"Storyboarding  Techniques  for  Slide/Tape  Presentations,”  and  "Authoring  Trainer 
and  Flight  Exercises."  Many  of  these  slide/tape  presentations  are  of  generic  as 
well  as  E-2  specific  utility.  Additional  instructional  materials  in  support  of  the 
Instructors-Under-Training  (IUT)  syllabus  included  an  Instructor's  Manual,  a 
Cockpit  Procedure  Trainer  (CPT)  Instructor's  Manual,  Pilot  and  NFO  Instructors 
Flight  Syllabi,  and  a  series  of  slide/tape  presentations  on  "WST  Operations." 
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IMPLEMENTATION  AND  EVALUATION 

E-2C  NFO  implementation  was  initiated  at  RVAW-120,  NAS  Norfolk  in 
November  1977.  Initially,  a  six-man  NFO  class  entered  the  ISD-based 
syllabus,  followed  by  a  three-man  FT  class  in  January  1978.  While  the 
NFO  class  entered  the  system  first,  their  additional  requirement  of  AIC 
training  allowed  the  FT  class  to  proceed  several  modules  ahead  of  them  in 
the  ISD-based  syllabus.  This  schedule  allowed  the  FT  class  to  serve  as 
subjects  in  a  formative  evaluation  concurrent  with  the  fulfillment  of  their 
training  requirements.  Despite  delays  in  production  due  to  problems  in 
obtaining  the  required  level  of  Navy  SME  support,  materials  were  available 
as  required.  Construction  of  the  NFO  Student  Learning  Center  was  completed 
ahead  of  schedule,  but  equipment  problems  with  the  tape  players  were  en¬ 
countered  on  a  continual  basis. 

E-2C  pilot  implementation  began  in  January  1978  with  an  E-2B-to-E-2C 
transition  class.  This  transition  class  provided  the  opportunity  for  a 
comprehensive  formative  evaluation  of  many  of  the  tape/slide  lessons  in  the 
ISD-based  syllabus.  Prior  to  the  first  full-syllabus  E-2C  pilot  class,  it 
was  determined  that  an  interim  syllabus  would  be  necessary  to  accommodate  a 
delay  in  the  delivery  of  the  OFT.  Accordingly,  greater  reliance  was  directed 
toward  the  CPT  and  the  aircraft  flights.  An  Interim  Flight  Syllabus  Guide 
and  an  Interim  CPT  Syllabus  Guide  were  developed  to  meet  these  requirements. 

A  pilot  class  was  then  convened  in  March  1978. 

E-2B  implementation  at  RVAW-110,  NAS  Miramar  began  in  June  1978.  In¬ 
structional  materials  for  the  E-2B  pilots  were  available  as  needed.  The 
impact  of  authoring  delays  In  the  E-2C  syllabus  (and  subsequent  E-2C-to~ 

E-2B  conversion  delays)  as  well  as  low  manpower  levels  of  E-2B  SME  resources 
at  that  squadron  resulted  in  a  temporary  reliance  on  classroom  vs.  individualized 
presentations  for  several  lessons.  An  interim  syllabus  was  also  required  to 
accommodate  the  lack  of  an  OFT  as  in  the  E-2C  implementation. 

Implementation  followed  the  procedures  prescribed  in  the  Implementation 
Plan,  the  Evaluation  Plan,  and  the  Instructor  Training  Course.  On-site  IPs 
and  training  analysts  worked  closely  with  the  Navy  ISD  Departments  where 
coordination  was  maintained  with  the  operations  departments.  Monthly  Progress 
Review  Meetings  proceeded  throughout  the  implementation/evaluation  period 
with  emphasis  on  provisions  for  long-term  revision  support. 


During  the  period  of  July  1978  through  July  1979,  contractor  support  was 
maintained  to  continue  the  summative  evlaluation  and  to  provide  revision 
consultation  and  production  support.  Data  collected  during  that  period 
documented  the  requirement  for  a  20%  revision  effort  to  "fine-tune"  the  system. 
These  revisions  were  of  an  iterative  nature,  covering  the  entire  range  of 
instructional  materials  and  procedures.  Additionally,  hardware  and  software 
modifications  in  the  E-2C  required  extensive  content  revisions  to  the  instructional 
materials.  This  was  particularly  true  of  the  NFO/FT  syllabus,  which  had  to 
accommodate  the  ”C-3"  software  change.  The  revised  NFO/FT  syllabus  is  depicted  in 
block  flow-chart  form  in  Appendix  B. 
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REVISION  CONSULTATION 

The  Phase  II  revision  plan  is  a  comprehensive  program  under  which  both 
formative  and  summative  revisions  are  documented  and  reviewed  for  inclusion 
in  both  Pilot  and  NFO/FT  syllabi.  Formative  revisions  were  first  documented 
during  the  Materials  Tryout  period  in  which  one  class  of  E-2B  transitioni 
pilots  and  one  class  of  FT’s  participated.  Discrepancies  which  were  discovered 
during  Materials  Tryout  were  documented  on  "Revision  Data  Sheets"  (Fig.  20)  ) 
for  both  audio-visual  and  printed  materials  discrepancies.  These  revision  data 
sheets  were  subsequently  reviewed  and  verified  by  Training  Development  Depart¬ 
ment  (TDD)  and  contractor  personnel.  Discrepancies  were  then  transcribed  to 
the  "Audio-Visual  Revision  Specifications"  (Figure  21).  The  revision  speci¬ 
fications  were  either  forwarded  to  the  contractor's  Buffalo,  N.  Y.  facility  for 
the  graphic  corrections  or  retained  on-site  for  narrative  and  printed  material 
corrections.  To  provide  for  the  interim  period  while  the  revisions  are  being 
completed,  "Errata  Sheets"  (Figure  22)  are  provided  with  each  lesson  to  identify 
pending  revisions.  In  most  cases,  slides  with  minor  discrepancies  were  not 
removed.  However,  if  the  discrepancy  was  evaluated  to  have  a  detrimental  effect 
on  the  instructional  value,  the  slide  was  then  removed  and  a  temporary  filler 
slide  was  installed  in  its  place. 

Because  of  the  overlap  between  the  formal  program  implementation  and  the 
audio-visual  production  schedule,  both  formative  and  summative  revision  data  were 
collected  within  the  same  time  frame  but  not  simultaneously.  The  "Formatlve- 
Summative  Review"  plan  is  shown  in  Figure  23.  In  accordance  with  the  plan, 
the  finished  slides  were  shipped  from  Buffalo  to  the  E-2C  Project  Office  in 
Norfolk,  Va.  Once  the  slides  were  received  in  Norfolk,  the  audio-visual  lessons 
were  prepared  for  review  and  Issued  to  TDD  for  subsequent  issue  to  and  evaluation 
by  Pilot/NFO  instructor  personnel.  During  the  review  process,  rough  discrepancies 
were  recorded  on  revision  data  sheets  and  then  returned  to  the  TDD  for  review  and 
verification.  Once  the  discrepancies  were  verified,  a  suggested  revision  was  also 
recorded  on  the  revision  data  sheet.  The  completed  revision  data  sheets  were 
returned  to  the  contractor  for  documentation  and  the  preparation  of  audio-visual 
revision  specif icetions.  It  should  be  noted  that  whenever  necessary,  attachments 
were  made  to  the  revision  specifications  to  allow  for  a  detailed  description  of 
the  correction  to  be  made.  The  graphic  revision  specifications  were  then  forwarded 
to  Buffalo  for  production  with  a  copy  of  the  specification  retained  in  Norfolk. 
These  epee if icetions  served  to  document  outstanding  discrepancies  and  were  cata¬ 
logued  by  lesson  on  a  VIDS  Board  "Revision  Status  File"  (Figure  24)  for  easy  ref¬ 
erence  end  identification.  Discrepancies,  once  corrected,  are  removed  from  the 
VIDS  Board  and  filed.  Prior  to  the  audio-visual  lessons  being  issued  to  the 
Pilot/NFO  Training  Departments,  an  errata  sheet  was  prepared  which  fully  described 
any  pending  revisions.  Here,  it  should  be  noted  that  for  those  revision  specifica¬ 
tions  retained  for  on-site  production,  Immediate  corrective  action  was  taken  when¬ 
ever  possible. 

Summative  evaluation  and  revision  data  is  collected  from  several  sources  once 
the  lessons  have  been  accepted  for  student  utilisation.  One  of  the  primary  sources 
for  summative  revision  data  is  feedback  from  the  "student  comment  section"  of 
tha  Student  Attitude  Surveys  (Fig.  25)  which  are  completed  by  the  students  for 
each  lesson.  These  surveys  are  then  reviewed  by  the  respective  Pilot  or  NFO 
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Type  of 


Lesson: 


Revision :  (check  boxes) 

Narrative:  □  quality  □  technical  □ 

Graphic:  □  quality  □  technical  □ 

Instructional  Matenal/Approach  □ 

Indicate  date  following  occurred: 

1.  Documented:  _ 

2.  Mailed:  - 

3.  Returned: _ 

4.  On-Site  Work  Comp: _ 

5.  Incorporated: _ 


Revlaion  Description: 


Fr.# _ 


Figure  21.  Audio  Visual  Specifications 
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LESSON# _  PILOT  □  NFO  Q 

SL'OE  #  DESCRIPTION  OF  CHANGE 
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STUDENT  ATTITUDE  SURVEY 


LESSON  V . 


□  PILOT 

CD  NFO 


!l  Errors  on  first  test: _ 

Time  required  to  complete 

lesson : _ 

(check  out  to  check  in) 


RANK/ RATE. 


STUDENT  CATEGORY: 


Q  ONE 
O  TWO 
Q  THREE 


OVERALL  LESSON  ASSESSMENT: 


(No  previous  AEW  experience) 
(Previous  experience;  not  in  model) 
(Previous  experience;  in  model) 


(Circle  number  of  choice  on  scale) 


5  Excellent 


General  Comments  and/or  recommendations: 


Figure  28.  Student  Attitude  Survey  Form 
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*r.^h'. A.'.CK  :  In  rel.it  ion  to  ay 
b  i nd  carter,  L  considered 
r.  ».*•  i  nst  rue  t  ion 


oN.ioYMt.NT :  The  instruction  was 


VI- jl'M:  The  amount  of  I  if or?ia t  ion 

-  i-'. -red  in  tiiis  lesson  was 


IMr.Kr.sr:  The  material  covered  is 


:  Tiu*  i  nst  rue  t  ion  in  this 

lesson  was 


PKhh  KKKN(.K :  If  given  a  choice  for 
future  instruction  I  would 


Qiihs  T  lu\S  :  The  instruction  allowed 
for 


No.  of 

Resp. 

\t  t  i  t  ude 

a  .  e  x  t  r  e  i  ie  :  y  r  e  1  e  .  a  a  t 

b •  relevant 

*  .  i  rr  - !  e-'t 

d.  ext  re.ii.*  i  "  irrelevant 

a.  extremely  enjoyable 

b.  enjwViSie 

r.  unenjoyaMe 

d.  o>;(  re:;vl;;  unen  joya b  1  e 

<i.  overwhelming  in  the  time 

a  1  lotted 

b.  kept  me  busy 

c.  easily  completed  in  time 

a  1  ior  t ed 

d.  c  oi.ip  i  et  eo  almost  immediately 

a.  extremely  Interesting 

b.  interesting 

c.  boring 

d.  extremely  boring 

i.  too  demand  i  ng 

b.  c ho i leng i ng 

c.  not  really  challenging 

d.  too  simple 

a .  like  to  take  more 

instruction  in  this  form 

b.  not  like  to  take  any 

instruction  in  this  form 
(it  b  chosen  fill  in  c) 

c.  Suggest 

form  for  this  instruction 

a.  opportunities  to  get  unclear 

points  clarified 

b.  not  enough  opportunities  to 

got  unclear  points  clarified 

Figure  25.  (Continued) 
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8.  ORGANIZATION:  The  concepts  In  the 
Instruction  of  the  material  In  this 
lesson  were 


9.  TESTS:  I  found  test 


10.  FEEDBACK:  Opportunities  for 
feedback  were 


11.  MATERIALS:  Audio/Visual  aids, 
Classroom  handouts  were 


a.  extremely  organized 

b.  organized 

c.  unorganized 

d.  completely  unorganized 

a.  covered  only  the 

Instructional  materials 

b.  covered  material  not  In¬ 

cluded  in  the  instruction 

a.  completely  adequate 

adequate 

c.  inadequate 

d.  completely  Inadequate 

a.  extremely  easy  to 

understand 

b.  understandable 

c.  difficult  to  understand 

d.  completely  confusing 

e.  if  c  and/or  d,  please 
specify 


Figure  26.  (Continued) 
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Training  personnel  and  then  returned  to  TDD  where  they  are  again  reviewed 
and  evaluated.  Potential  problem  areas  and/or  discrepancies  are  then  verified 
through  additional  lesson  review  by  TDD  personnel.  Discrepancies  that  are 
verified  are  again  documented  on  revision  data  sheets  and  given  to  the  contractor's 
on-site  personnel  for  documentation  and  processing.  Other  sources  of  evaluation 
data  include  reviews  of  student  performance  records  by  both  TDD  and  contractor 
personnel  to  further  assist  in  the  evaluation  of  internal  and  external  program 
validity.  This  is  partially  accomplished  through  a  review  of  student  record 
sheets  to  determine  both  time  and  number  of  attempts  required  to  reach  criteria 
for  the  cognitive  and  practice  lessons.  External  validity  is  further 
evaluated  once  a  sufficient  number  of  ISD-Trained  students  have  been  placed  in 
fleet  squadrons  to  permit  the  completion  of  a  valid  survey  in  the  form  of  Fleet 
Feedback  questionnaires. 
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SECTION  X? 


RESULTS  AMD  OOMCLUSXOMS 


SUMMATIVE  EVALUATION 

Data  collection  for  the  summative  evaluation  began  with  implementation  and 
will  continue  on  a  long-term  quality  control  heals.  Internal  quality  control 
la  aonltored  by  records  of  student  performance  as  described  in  the  section 
entitled  "curriculum, "  by  'Student  Attitude  Surveys"  completed  for  each  lesson, 
and  by  the  end-of-course  "RVAW  Critique."  External  validation  la  determined 
by  response  to  the  "Fleet  Feedback"  questionnaire. 

Table  4  describes  the  magnitude  of  the  total  ISD  effort  with  a  breakdown 
of  the  number  of  Tasks,  Task  Elements,  Behavioral  Objectives,  and  Lessons  by 
Media.  Additionally,  the  number  of  test  items  and  performance  measures  are 
Indicated. 

Tables  5  and  6  describe  the  pilot  and  NFO/FT  syllabi  before  and  after 
Implementation  of  the  ISD-based  syllabi.  These  tables  are  presented  for 
comparative  purposes  in  evaluating  the  ISD-based  training  system. 

Tables  7  and  8  summarize  the  academic  performance  of  the  ISD  graduates  on  the 
lesson  tests  and  NATOPS  exams  respectively.  The  academic  norms  for  learning  time 
are  criterion  referenced  and  inclusive  of  testing  and  remediation.  The  mean 
error  rates  are  for  first  attempt  on  end-of-lesson  tests.  NATOPS  exam  scores 
indicate  a  high  level  of  mastery  in  the  relevant  areas. 

Figures  26  and  27  illustrate  the  results  of  the  "RVAW  Critiques"  which 
are  completed  by  graduates  of  the  ISD-based  training  program  prior  to  check-out. 
With  the  exception  of  a  few  squadron-related  functions  which  are  beyond  the 
purview  of  ISD,  the  mean  ratings  indicate  a  significantly  favorable  assessment. 

Figures  28  and  29  illustrate  the  graduates’  appraisal  of  the  ISD-based  E-2 
Training  from  the  perspective  of  performance  in  the  fleet.  Each  graduate  is 
asked  to  complete  a  "Fleet  Feedback"  survey  after  completing  a  minimum  of  six 
months  of  operational  duty.  Commanding  Officers  are  also  asked  to  rate  each  of 
the  ISD  graduates.  The  results  indicate  a  highly  positive  trend,  especially  in 
the  Commanding  Officer’s  assessment. 

Finally,  Figure  30  summarizes  student  attitude  responses  to  the  individual 
lessons.  The  composite  data  indicates  "favorable”  or  "very  favorable"  acceptance 
In  each  of  the  eleven  areas  of  the  "Student  Attitude  Survey." 

PROJECTED  SAVINGS 

The  average  course  length  for  the  ISD-based  pilot  syllabus  is  scheduled 
to  be  approximately  16  calendar  weeks.  This  represents  a  reduction  of  four 
weeks  from  the  previous  syllabus.  The  NFO  syllabus  has  been  reduced  from 
33  weeks  to  approximately  26  weeks.  These  reductions  result  from  efficiencies 
in  scheduling  as  well  as  from  actual  decreases  in  syllabus  hours.  Of  perhaps 
even  greater  importance  to  cost-effectiveness,  the  basic  pilot  syllabus 
(with  the  OFT)  prescribes  a  substantial  (20Z)  reduction  in  flight  time  through 
the  ISD  approach  to  training. 
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TABLE  4.  ISO  COMPONENT  DATA  FOR  E-2  PILOT  AND  NFO/FT  (MAY  1978) 


ITEM 

PILOT 

NFO/FT 

TASKS 

84 

430 

TASK  ELEMENTS 

1105 

1528 

BEHAVIORAL  OBJECTIVES 

159 

465 

LESSONS  BY  MEDIA: 

TAPE/SLIDE 

57 

63 

CLASSROOM 

7 

14 

TACTICS  TRAINER 

1 

32 

OPERATIONAL  FLIGHT  TRAINER 

12 

- 

COCKPIT  PROCEDURES  TRAINER 

10 

- 

AIRCRAFT  (GROUND) 

- 

3 

AIRCRAFT  (FLIGHTS) 

14 

9 

APPROXIMATELY  2400  TEST  QUESTIONS  AND  1800  PERFORMANCE 
MEASURES  (PRACTICE  8i  SORTIE  SCENARIO) 
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TABLE  5.  PILOT  SYLLABUS  HOURS  BEFORE  AND  AFTER 

IMPLEMENTATION  OF  THE  I SD- BAS ED  SYLLABUS* 


BEFORE  ISD 

AFTER  ISD 

ACADEMIC  TRAINING 

128.0 

79.5 

COCKPIT  PROCEDURES  TRAINER 

13.0 

28.0** 

OPERATIONAL  FLIGHT  TRAINER 

- 

37.0 

TACTICS  TRAINER  (15F8) 

- 

2.5 

FLIGHT  TRAINING  (E-2) 

45.0 

36.0 

TRAINING  SYLLABUS  TOTAL 

186.0 

183.0 

♦Figures  do  not  include  the  Field  Carrier  Landing 
Practice  (FCLP)  and  Carrier  Qualification  (CQ)  flight 

hours,  which  can  only  be  prescribed  on  an  individual 

basis. 

♦♦Figures  include  CPT  and  CPT/CRL  hours. 


( 


l 


X 
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TABLE  6.  NFO/FT  SYLLABUS  HOURS  BEFORE  AND  AFTER 

IMPLEMENTATION  OF  THE  ISD-BASED  SYLLABUS* 


ACADEMIC  TRAINING 

TACTICS  TRAINER  (15F8) 

PREFLIGHT  TRAINING 
( E—  2  GROUND-BASED) 

FLIGHT  TRAINING 
(E-2  AIRBORNE) 

TRAINING  SYLLABUS  TOTAL 

*Figures  do  not  include  AIC 


BEFORE  ISD 

AFTER  ISD 

229.0 

138.0/139.0 

113.  5 

66. 0 

7.0 

6.0 

23.0 

23.0 

372.5 

233.0/234.0 

Training. 
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TABLE  7.  ACADEMIC  NORMS:  E-2  ISD  GRADUATES 


CURRICULUM 

LEARNING  TIME 
(HOURS) 

PERCENT  CORRECT 
(FIRST  ATTEMPT) 

PILOT 

125-130 

80  -  84% 

NFO/FT 

100-105 

80  -  84% 
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TABLE  8.  NATOPS  EXAM  SCORES:*  E  2  ISD  GRADUATES 


OPEN 

BOOK 

CLOSED 

BOOK 

ORAL 

EVALUATION 

FLIGHT 

PILOT 

X  = 

3.89 

3.77 

3.71 

NFO/FT 

X  = 

3.94 

3.62 

3.67 

3.74 

*  4.0  SCALE 
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Mori* 
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Figure  27.  NFO/FT  RVAW  Critique:  ISD  Graduates 
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Figure  29.  NFO/FT  Fleet  Feedback:  ISD  Graduates 
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CONCLUSIONS 
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TRAINING  DEVELOPMENT  METHODS.  The  contractor's  methodology  appears  to  have 
successfully  integrated  the  systematic  procedures  of  ISD  with  the  professional 
judgments  of  instructional  psychologists  and  subject  matter  experts.  The  frame¬ 
work  established  by  the  contractor's  analysis  and  design  studies  in  Phase  I 
has  directed  the  Phase  II  development,  but  in  an  iterative  fashion.  In  many 
cases,  subject  matter  considerations  and  squadron  resources  and  constraints  have 
suggested  modifications  to  the  originally-proposed  ISD-based  syllabus  and  lesson 
specif ications.  These  changes  have  been  incorporated  whene/er  justified  by 
appropriate  revisions  to  the  analytical  data  bases  (i.e.  Task  Analysis,  Behavioral 
Objectives,  Method-Media  Selection).  The  resulting  curriculum  is  therefore 
completely  ISD-based  but  also  enhanced  by  a  responsive  system  of  Navy  SME  input. 

TRAINING  DEVELOPMENT  IMPLEMENTATION.  The  utilization  of  Navy  SMEs  for  the 
authoring  of  training  materials  under  the  direction  of  contractor  IPs  has 
been  successfully  integrated  into  the  E-2  ISD  program.  The  success  of  this 
approach  is  closely  tied  to  the  teaming  of  Navy  and  contractor  technical 
management  with  mutual  understanding  of  areas  of  expertise  and  responsibility. 

Under  this  direction,  highly-skilled  SMEs  can  be  very  effective.  Naval  officers 
in  the  E-2  community  have  proven  to  be  readily  adaptable  to  the  ISD  tasks  through 
SME  author  training  provided  by  documented  guidelines  and  on-the-job  experience. 
Additionally,  attention  to  the  affective  domain  in  their  transition  to  the  ISD 
approach  appears  to  have  been  a  crucial  factor  in  their  effectiveness.  Of  equal 
importance,  the  input  of  operational  squadron  personnel  has  been  a  prime 
consideration. 

TRAINING  DEVELOPMENT  RESOURCES.  The  availability  of  Navy  SMEs  for  work  on  the 
ISD  tasks  has  been  a  critical  problem.  Despite  a  concerted  squadron  commit¬ 
ment  to  the  effort,  production  was  delayed  due  to  problems  in  this  area.  Correc¬ 
tive  actions  were  taken  early  enough,  however,  to  meet  the  needs  of  the  effort. 
Accordingly,  additional  man-power  resources  were  incorporated  through  temporary 
duty  assignments  of  personnel  assigned  by  the  Wing,  part-time  utilization  of 
squadron  personnel  outside  the  TDD/ISD  Departments,  and  contractor  SME  support. 

It  appears  that  the  use  of  contractor  SMEs  was  a  critical  factor  to  the  successful 
implementation.  Therefore,  it  is  recommended  that  future  programs  follow  such  an 
approach  to  fill  a  significant  portion  of  the  authoring  task.  Nevertheless,  the 
contractor's  on-site  conduct  of  the  effort  and  the  participation  of  Navy  SMEs  for 
direction,  review,  and  as  much  authoring  as  can  be  feasibly  accommodated,  appears 
to  be  warranted. 

RECOMMENDED  ADJUSTMENTS  TO  SPECIFICATIONS.  While  the  E-2  ISD  effort  was  not 
directed  by  the  MIL-T-29053  specification,  the  contractor  has  incorporated  its 
input  whenever  feasible.  Three  areas  within  that  specification  and  its  correc- 
sponding  DIDs  appear  to  be  in  need  of  revision.  The  first  suggested  revision 
involves  the  need  for  detailed  guidelines  on  the  format  and  style  for  instructional 
materials  production,  particularly  in  the  audio-visual  domain.  The  second  suggested 
revision  involves  the  DID  entitled  "Quality  Control  Report"  (UDI-H-25729) .  It 
is  recommended  that  the  reporting  format  for  revision  documentation  be  modified  to 
reflect  depth  as  well  as  quantity.  The  third  suggested  revision  involves  the  DID 
entitled  "Training  Program  Work  Report"  (UDI-H-25731).  It  is  recommended  that 
redundancies  in  the  ISD  and  Implementation  sections  be  eliminated,  and  that  a 
formal  section  on  "Evaluation  Data"  be  incorporated. 
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OBSERVATIONS.  The  E-2  ISD  program  appears  to  have  made  a  significant  contribution 
to  the  ISD  methodology  in  the  area  of  developing  documentation  for  "hands-on" 
lessons.  Instructor  and  Student  Guides  have  been  developed  in  a  manner  which 
relates  directly  to  the  Behavioral  Objectives.  Corresponding  evaluation  scales 
which  provide  the  instructor  with  the  opportunity  to  teach  performance  skills  as 
well  as  assess  student  performance  have  also  been  successfully  incorporated.  It 
appears  that  these  provisions  have  greatly  enhanced  the  syllabi. 

A  second  observation  applies  to  audio-visual  materials.  It  appears  that  the 
visual  fidelity  employed  in  the  E-2  tape/slide  presentations  has  greatly  facilitated 
the  learning  process  by  forming  the  cornerstone  of  a  building  block  approach  to 
psychomotor  skills  development.  The  contractor  has  developed  many  graphic  tech¬ 
niques  to  make  this  approach  cost-effective. 
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SECTION  V 


RESOURCE  UTILIZATION 


Figures  31  and  32  present  task  expenditure  data  for  the  E-2  ISD  program. 

For  each  task,  contractor  data  includes  category  of  assigned  personnel  and 
manhours  expended.  Data  provided  on  Navy  support  include  manhours  expended  for 
supervisors,  NFOs  and  pilots. 

PROBLEM  AREAS 

Several  problems  developed  which  had  a  significant  impact  on  resource 
utilization  and  expended  contractor  funds. 

SME  time  proved  to  be  a  serious  problem  for  the  Navy.  This  was  the 
result  of  a  contractual  requirement  for  more  SME  personnel  than  were  available, 
resulting  in  slippage  in  the  storyboard/lesson  guide  production  schedules.  A 
solution  to  the  problem  was  the  hiring  of  additonal  SMEs  by  the  contractor 
and  temporary  duty  assignments  for  Naval  personnel  from  outside  the  Squadrons 
and  the  Wing.  These  SMEs  supported  the  storyboard/lesson  guide  production  as 
well  as  authored  classroom  materials,  student  handouts,  and  tests. 

A  long-term  delay  in  the  delivery  of  the  E-2C  OFT  necessitated  the  development 
of  an  interim  pilot  syllabus  relying  more  heavily  on  aircraft  and  Cockpit  Procedures 
Trainer  usage.  This  interim  flight  syllabus  was  not  part  of  the  original  program. 

The  absence  of  definitive  guidelines  on  the  artistic  approach  for 
tape/slide  lessons  resulted  in  many  iterations  of  artwork  before  a  technique 
was  developed  that  was  acceptable  to  the  Navy  and  the  contractor. 

The  hardware  change  from  the  APS-120  to  the  APS-125  required  develop¬ 
ment  of  two  separate  tracks  in  the  NFO  syllabus,  and  the  addition  of  several 
lessons  which  were  not  part  of  the  orginal  program. 

A  final  problem,  and  one  that  will  impact  on  the  program  in  the  future, 
was  the  rapid  frequency  of  software  changes  in  the  E-2C  aircraft  itself 
and  in  the  15F8  Tactics  Trainer. 

REQUIRED  EQUIPMENT  AND  FACILITIES 

Government-furnished  office  space  and  equipment  was  provided  for 
the  contractor  at  NAS  Norfolk.  Personnel  assigned  to  this  office  included 
three  instructional  psychologists,  one  training  analyst,  a  technical  writer, 
and  secretarial  and  typing  support. 
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CONTRACTOR  NAVY 


PERSONNEL  MANHOURS  ISD  MANHOURS  EXPENDED 


TASKS 

ASSIGNED  EXPENDED 

SUPERVISOR 

NFO 

PILOT 

CATEGORY 

PROGRAM  PLANNING 

2326 

1320 

323 

428 

COURSE  REVIEW 

244 

195 

342 

92 

LESSON  SPEC  REVIEW 

* 

126 

60 

293 

135 

FLESHING  LESSON  SPEC 

INSTRUCTIONAL 

PSYCHOLOGISTS, 

190 

67 

464 

323 

STORYBOARDING 

)  TRAINING  ANALYSTS, 

AND  ADMINISTRATIVE 

2200 

243 

1910 

2287 

TEST  DEVELOPMENT 

SUPPORT 

214 

90 

372 

222 

MATERIAL  TRYOUT 

990 

86 

120 

76 

FORMATIVE  REVISION 

16 

106 

466 

62 

INSTRUCTOR  TRAINING 

J 

540 

GRAPHICS  SUPPORT: 

PHOTOGRAPHER 

PHOTOGRAPHER 

2020 

PHOTOMECHANICS 

PHOTOMECHANICS 

815 

ILLUSTRATOR 

ILLUSTRATOR 

9173 

SPECIALTY  TYPIST 

TYPIST 

123 

REPORT  REPRO 

REPORT  REPRO 

244 

TOTAL: 

19221 

2167 

4290 

3625 

SECRETARIAL,  NARRATION  AND  MISCELLANEOUS  PURCHASED  SERVICES 
NOT  INCLUDED  ABOVE. 


Figure  31.  Resource  Utilization  Log  (October  31,  1976  -  May  28,  1978) 
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The  remainder  of  the  requirements  for  equipment  and  facilities  sup¬ 
port  was  necessary  for  audio-visual  production.  Contractor-provided  equipment 
included:  furniture  and  art  supplies  for  seven  illustrators;  35mm  slide  camera 

slide  duplicating  and  photography  processing  equipment  and  chemicals;  and  re¬ 
port  reproduction  equipment. 


Task  Duration  for  Phase  II  E-2  ISP  Program 


Instructional  Materials  Development 

$ 

21 

months 

Aviation  Training  Support  System  (ATSS) 

12 

months 

Evaluation  Planning 

2 

months 

Implementation  Planning 

2 

months 

Instructor  Training  Course 

5 

months 

Implementation  and  Evaluation 

6 

months 

Revision  Consultation 

7 

months 

Task  Duration  for  Follow-on  Revision* 


Extension  of  Summative  Evaluation 
Revision  Specifications 
Graphics/Narration  Revision 
IUT  Syllabus 

*Includes  overlap  with  Phase  II  and  "C-3"  NFO/FT  Revision 


18  months 
18  months 
18  months 
13  months 
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APPENDIX  A 


AUDIO-VISUAL  PRODUCTION 


The  audio-visual  production  techniques  described  in  this  Appendix 
were  developed  as  a  sub-task  within  Phase  II  of  the  E-2  ISD  effort  on  Navy 
Contract  N61339-77-C-0003.  The  major  task  of  this  program,  Instructional 
Materials  Development,  was  comprised  of:  lesson  specification  reviews,  de¬ 
tailing  of  lesson  specifications,  authoring  and  test  development,  production, 
and  materials  tryout  and  formative  revision.  Although  the  production  sub-task 
also  included  typing,  printing,  and  duplicating  support  for  "hard  copy"  train¬ 
ing  materials,  only  the  complex  audio-visual  processes  will  be  addressed  here. 

Detailing  of  the  lesson  specifications  developed  by  the  contractor 
on  Phase  I  of  the  E-2  ISD  program  was  primarily  the  function  of  Navy  and  con¬ 
tractor  subject  matter  experts  (SMEs)  in  Norfolk.  These  detailed  lesson  speci¬ 
fications  were  used  aa  guidelines  for  storyboarding  the  tape-slide  lessons. 

For  each  teaching  point,  the  detailed  storyboard  process  provided  a  frame-by- 
frame  description  of  the  visual,  corresponding  narrative,  and  related  test 
items.  The  result  was  several  thousand  pages  of  documentation  for  input  to 
the  audio-visual  production  process. 

PRODUCTION  QUALITY  CONTROL 

After  extensive  technical  review  of  the  storyboards  by  the  SMEs  and 
TDD  in  Norfolk,  each  lesson  was  reviewed  by  contractor  media  specialists  in 
Buffalo.  This  review  process  standardized  the  visual  material  presentation 
and  amplified  the  productions  details  for  the  illustrators. 

Because  approximately  50,000  slides  were  generated  during  the  pro¬ 
duction  effort  of  Phase  II,  a  "tracking"  procedure  was  developed  to  account 
for  each  frame  of  instruction.  As  each  lesson  was  reviewed  by  media  specialists, 
a  log  sheet  was  completed  for  every  frame  indicating  the  type  of  visual  support 
required  (e.g. ,  contractor-generated  art,  photograph  or  direct  pick-up 
from  existing  visual  material). 

Dally  interaction  between  the  media  specialists  and  illustrators 
provided  answers  to  artistic  approach  questions  as  they  occurred,  and  tech¬ 
nical  direction  was  received  from  Norfolk.  Review  of  finished  artwork  also 
occurred  daily,  allowing  for  immediate  feedback  to  the  illustrator  on  artistic 
approach  and  technical  material.  After  review  and  correction,  each  piece  of 
finished  art  was  labeled  with  lesson  and  frame  numbers  and  then  photographed. 

TWo  original  slides  were  made  for  each  piece  of  contractor-generated 
art.  After  developing  and  mounting,  one  of  these  slides  was  retained  for  the 
master  file.  The  second  original  slide  became  the  master  from  which  the  nec¬ 
essary  duplicates  were  made.  Completed  slides  were  labeled,  collated,  and 
filed  in  Carousel  slide  trays. 
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In  determining  the  artistic  approach  for  visual  production,  the  con¬ 
tractor  was  directed  to  utilize  as  many  of  the  existing  slides  as  possible  to 
effect  cost  savings.  Incorporating  revised  technical  information  and  new 
material  required  the  development  of  processes  that  would  produce  artwork  that 
was  compatible  with  the  quality  and  techniques  of  the  existing  material. 

The  bulk  of  the  original  art  was  produced  in  two  ways.  For  the  first 
technique,  NFO  panels  and  gauges  were  rendered  using  an  airbrush  technique 
which  resulted  in  a  realistic  representation  of  the  material.  In  the  second 
technique,  a  multi-step  "Disney”  approach  was  used  to  create  realistic  pilot 
panels  and  gauges.  These  two  techniques  are  discussed  in  greater  detail  in 
the  following  paragraphs. 

AIRBRUSH  RENDERINGS 

The  first  step  in  the  airbrush  rendering  process  is  tracing  the 
panel  and  gauge  faces  from  existing  engineering  drawings.  The  tracings  are 
"painted"  with  an  airbrush,  filling  in  a  solid  background  first,  then  adding 
shadows  where  switches  and  dials  are  to  appear.  Color  photographs  of  all 
panels  and  gauges  provide  the  necessary  detail  for  accurate  color  matching 
of  console  faces  and  the  various  indicator  lights. 

Completed  airbrushed  panels  and  gauges  are  photographed  to  provide 
bases  on  which  to  indicate  various  switch  positions  and  dial  readings.  Using 
a  rub-on  transfer  technique,  switches  and  dials  are  applied  to  the  photographed 
panel,  and  gauge  needles  are  positioned  where  necessary.  Great  quantities  of 
original  art  can  be  rapidly  generated  by  simply  changing  a  dial  or  switch 
position  in  this  way. 

An  example  of  a  panel  completed  using  an  airbrush  technique  and 
rub-on  transfers  for  labeling  switches  is  shown  in  Figure  A-l.  Figure  A-2  is 
an  example  of  switches  applied  with  a  rub-on  transfer. 

"DISNEY"  APPROACH 

The  first  step  in  the  "Disney"  approach  is  to  trace  the  panel  and 
gauge  faces  from  existing  engineering  drawings.  At  this  point,  only  the  out¬ 
line  is  traced,  and  switches  and  dial  locations  are  added.  The  panel  lettering 
is  sized  by  the  typist. 

In  the  second  step  of  the  process,  the  original  material  produced  in 
the  preceding  step  is  photographed.  This  results  in  a  print  that  allows  easy 
application  of  the  rub-on  transfer  material. 

In  step  three,  switches  are  applied  in  the  correct  position  using 
rub-on  transfer  material.  Only  necessary  touch-ups  are  made  at  this  point. 
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Figure  A  1.  EXAMPLE  OF  AIRBRUSHED  RENDERING 
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Figure  A-2.  EXAMPLE  OF  RUB-ON  SWITCH  TECHNIQUE 
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In  step  four,  a  film  positive  is  made  of  the  artwork,  and  an  overlay 
of  the  lettering  is  added.  The  back  of  the  film  positive  is  painted,  and 
shadows  are  added,  resulting  in  a  realistic  representation. 

In  step  five,  all  layers  of  material  are  assembled  on  a  layout  board 
and  double-checked  for  technical  accuracy.  After  the  board  is  labeled  with 
lesson  and  frame  numbers,  the  finished  art  is  ready  for  the  camera. 

In  the  final  step  of  this  process,  step  six,  the  finished  art  is 
photographed  as  an  original  slide.  Examples  of  each  step  of  this  process  can 
be  seen  in  Figures  A-3a  -  A-3f. 

MODIFICATION  OF  EXISTING  SLIDES 

As  mentioned  earlier,  training  material  already  existed  from  a  pre¬ 
vious  E-2  effort.  Whenever  possible,  this  material  was  used  without  modifica¬ 
tion.  When  teaching  points  called  for  visual  support  which  required  a  minor 
change  to  artwork  already  in  existence,  a  print  was  made  from  the  appropriate 
negative.  Necessary  modifications  were  made  to  the  print  using  any  of  the 
previously-described  techniques,  and  the  revised  photograph  was  mounted  on  an 
illustration  board  and  photographed  as  an  original  slide. 

AUDIO  PRODUCTION 

Because  audio  production  was  handled  in  Norfolk,  completed  tape/slide 
lesson  packages  were  supplied  to  the  Navy  in  two  ways.  Slide  trays  for  use  in 
RVAW-120  implementation  were  sent  directly  to  Norfolk  from  the  contractor's 
Buffalo  facility.  In  Norfolk,  each  slide  tray  was  checked  out  with  the  appro¬ 
priate  audio  tape  before  the  tray  was  used  by  students.  For  RVAW-110  implem¬ 
entation,  completed  audio  tapes  were  sent  to  Buffalo  from  Norfolk,  where  the 
tapes  were  checked  out  with  the  appropriate  slide  trays  and  then  forwarded  to 
San  Diego. 

COMMENTS 

Significant  cost  savings  were  realized  by  use  of  the  art  techniques 
described  in  this  Appendix.  Although  the  airbrush  rendering  and  "Disney”  ap¬ 
proaches  have  significant  start-up  time  in  their  first  step,  the  greatest  savings 
occur  once  all  the  original  panels  and  gauges  have  been  inked  and  realistically 
colored.  From  that  point  on,  multiple  photographs  of  the  original  art  can  be 
made  in  the  different  sizes  required  to  produce  the  necessary  bases  for  each 
frame  of  instruction. 

A  significant  saving  in  illustrator  time  was  realized  by  use  of  the 
rub-on  transfer  techniques  for  switches,  dials,  and  lettering.  Changes  in 
switch  positions  could  be  handled  in  a  matter  of  a  few  minutes,  without 
having  to  go  through  the  time-consuming  process  of  generating  entire  pieces 
of  original  art  from  "scratch." 
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Figure  A-3a.  "DISNEY"  APPROACH:  STEP  1 
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Figure  A-3d.  "DISNEY"  APPROACH:  STEP  4 
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APPENDIX  B 

NFO  SYLLABUS  REVISION  (C3) 

The  recent  incorporation  of  software  changes  into  the  E-2C  NFO  Training 
Syllabus  is  a  result  of  the  implementation  of  tape  C3/A3.  This  new  software 
tape  represents  a  significant  change  to  the  various  computer  subprograms  which 
make  up  the  overall  E-2C  Tactical  Program.  Although  the  basic  function  of  the 
subprograms  is  virtually  unchanged,  the  display  of  data  and  operator  interface 
functions  have  changed  significantly.  In  addition  to  the  32  Categories  and 
Functions  which  were  deleted,  approximately  6  new  Categories  and  19  Functions 
were  added  and  111  existing  functions  were  either  moved  to  other  existing 
categories  or  in  some  other  way  modified.  The  restructuring  of  the  Categories 
and  Functions  necessitated  the  generation  of  24  completely  new  lessons  directly 
related  to  the  C3/A3  tape  as  well  as  the  modification  of  8  additionally  existing 
lessons. 

As  a  result  of  the  C3/A3  software  tape  change,  the  NFO  Training  Syllabus 
has  undergone  major  revision  in  the  areas  of  subject  matter  and  lesson 
sequencing.  Although  the  number  of  lessons  and  course  length  remain  relatively 
unchanged,  the  restructuring  of  the  syllabus  provides  a  more  orderly  presentation 
of  subject  matter  in  relation  to  the  Integrated  Flight  Syllabus3.  Additionally, 
the  AIC  Course  has  been  moved  from  the  initial  portion  of  the  syllabus  so  that 
it  now  takes  place  at  a  point  where  the  student  has  already  completed  Flight  #4 
in  the  aircraft  and  all  of  the  basic  software  lessons.  Following  the  AIC  Course, 
the  Student  then  re-enters  the  basic  syllabus  and  proceeds  to  the  more  complex 
lessons,  flights,  and  Tactics  Trainer  exercises. 

aThe  Integrated  Flight  Syllabus  was  an  interim  syllabus  change  which  was 
instituted  after  original  NFO  Training  Syllabus  implementation. 
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